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ABSTRACT 


The  appropriateness  of  a  multidimensional  scaling  tech- 
nigue  (MDS)  in  attempting  to  quantify  students  perceptions 
of  teacher  performance  is  investigated  in  the  following 
study.  Data  collected  on  an  interactive  computer  survey  and 
from  Student  Opinion  Forms  (SOF's)  are  used  to  determine  if 
satisfactory  linear  relationships  for  teacher  performance 
exist.  Multiple  linear  regression  and  factor  analysis 
attempt  to  identify  what  appear  to  be  the  most  important 
characteristics  in  instructor  performance  according  to  the 
perceptions  of  a  control  group.  Spatial  plots  are  created 
reflecting  these  perceptions. 
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Z.  IHTBQDDCTION 


1.  PUBPOSB 

The  purpose  of  this  thesis  is  twofold: 

1.  To  develop  methodology  that  will  help  discover  the 
important  characteristics  of  instructor  performance 
as  perceived  by  each  student  group; 

2.  To  develop  user  friendly  software  compatible  with  our 

IBM  3033  system  that  facilitates  the  data  collection 
and  processing  in  support  of  the  foregoing.  . 

The  methods  designed  herein  request  proximity  data  or 
similarity /dissimilarity  data  on  pairwise  combinations  of 
professors  in  the  Operations  Research  department.  Also,  the 
respondents  are  requested  to  provide  ratings  on  several 
’bipolar  scales*  of  suggested  instructor  characteristics. 
The  methods  for  discovering  the  dimensions  or  relationships 
that  appear  to  characterize  the  professors  utilize  several 
statistical  tools  including  multidimensional  scaling, 
regression  analysis,  factor  analysis  and  cluster  analysis. 
The  data  used  in  the  analysis  comes  from  those  students  in 
the  Operations  Research  curriculum  graduating  in  March  of 
1985. 

An  interactive  computer  survey  is  designed  to  query  the 
students  on  their  perceptions  of  the  teaching  effectiveness 
of  the  instructors.  Thus,  data  is  input  by  the  students 
during  a  twenty  minute  session  on  the  3278  terminal. 
Initially  the  student  links  to  the  software,  and  when 
finished,  transmits  his  responses  to  a  central  file. 
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B.  SUBJECT 


The  subject  of  this  thesis  deals  with  quantifying 
perceptions.  Specifically,  ve  would  like  to  obtain  an 
explanation  as  to  why  students  perceive  instructors  as  being 
similar  or  dissimilar  and  discover  the  dynamic  factors  that 
a  particular  class  uses  to  discriminate  among  instructors. 
The  multidimensional  scaling  technique  uses  the  information 
from  the  survey  to  create  a  'multidimensional  map'  of 
points.  Each  point  represents  the  coordinate  position  of 
the  objects  under  investigation,  in  our  case,  professors. 
Once  this  spatial  plot  is  produced,  it  remains  for  the 
researcher  to  discover  those  factors  that  appear*  to  cause 
the  structural  relationships. 

Multiple  regression  analysis  and  factor  analysis  are  two 
techniques  commonly  used  to  describe  linear  relationships 
among  dependent  and  independent  variables.  Each  method  is 
given  consideration  here  in  attempting  to  interpret  the 
spatial  plot  produced  by  the  multidimensional  scaling 
program,  KYST  [Hef.  1].  Additionally,  cluster  analysis  is 
used  to  group  the  professors  into  disjoint  clusters.  This 
cluster  information  is  presented  to  the  students  during  exit 
interviews  to  help  guide  the  researcher  in  his  attempt  to 
find  the  underlying  relationships. 

C.  SCOPE 

This  thesis  is  presented  in  four  major  chapters 
excluding  the  introduction.  The  second  chapter  describes  in 
detail  the  background  of  multidimensional  scaling  methods. 
A  brief  account  of  regression  analysis,  factor  analysis  and 
cluster  analysis  is  given  as  well.  The  third  chapter  speci¬ 
fies  the  means  by  which  the  data  was  collected  in  the  inter¬ 
active  survey.  Chapter  IV  emphasizes  the  analysis  of  the 
data.  Finally,  Chapter  V  provides  a  summary  of  the  salient 
points  determined  in  the  study. 
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D.  1  BB3EF  SOHHABT 


A  four  dimensional  interpretation  was  emphasized  in 
describing  the  data  obtained  from  the  computer  survey  and 
the  SOF  forms.  These  four  factors  included,  1)  a  student- 
instructor  interaction  effect;  2)  the  degree  to  which  a 
professor  was  perceived  as  being  organized  or  prepared  for 
class;  3)  a  combined  effect  of  grading  policy,  effort 
required  outside  class  and  pace  of  the  course;  and  4)  a 
composite  effect  combining  class  size  and  the  degree  to 
which  a  course  relied  upon  prerequisites.  A  high  correla¬ 
tion  appears  to  exist  among  those  bipolar  scales  used  in  the 
current  SOF  form.  Further  investigation  hopefully  will  lead 
to  discovering  other  factors  that  will  help  describe  teacher 
performance.  The  results  obtained  in  this  study  are  not 
meant  to  be  predictive  but  explanatory.  The  value  associ¬ 
ated  with  an  instructor  for  each  characteristic  may  be 
regarded  as  his  score  on  that  dimension.  Thus,  rankings  of 
instructors  by  characteristics  are  possible. 


II.  background 


1.  WHAT  IS  HOLTIDIHEISIOMAL  SCALING? 

Multidimensional  scaling  involves  the  problem  of 

depicting  n  points  in  multidimensional  space  such  that  the 
interpoint  distances  correspond  in  some  manner  to  measured 
proximity  data  [  Bef .  2:  p.  1].  The  proximity  data  can  be 
similarities,  dissimilarities,  correlation  coefficients  or 
any  other  measure  of  association  as  perceived  by  a  set  of 
judges  participating  in  an  experiment.  Multidimensional 
scaling  techniques  attempt  to  produce  the  structure  or 
interrelationships  among  the  n  objects  by  assuming  a  direct 
correspondence  between  the  measured  proximity  data,  or 

dissimilarity  data  ,  in  our  case,  and  the  interpoint 

distances  d-,.*  .  Several  choices  of  multidimensional  sealing 
exist,  the  difference  being  the  assumed  relationship  between 
and  d^‘  .  The  ultimate  product  of  multidimensional 

scaling  (HDS)  will  be  a  spatial  map  that  displays  the  asso¬ 
ciation  between  the  n  objects  under  investigation.  MDS  has 
found  considerable  application  in  the  social  sciences, 
particularly  in  the  realms  of  psychology,  sociology, 
economics  and  education. 

A  significant  point  worth  pursuing  is  this  idea  of 
correspondence  between  the  proximity  data  -f.j  ,  and  the 
distance  data  d  k*  .  A  fairly  simple  method  of  analyzing  a 
possible  relationship  would  be  to  observe  a  scatter  plot 
similar  to  the  one  given  in  Figure  2.1  .  The  vertical  Y 
axis  of  the  scatter  plot  contains  the  measured  dissimilari¬ 
ties  iS.^,  while  the  horizontal  X  axis  shows  the  corresponding 
distances,  d.-  ,  computed  from  the  derived  set  of  character¬ 
istic  vectors. 


'?  ‘viva  Aityviiwssia 
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Scatter  Plot  of  Distance  ts-  Dissimilarity  Data 


Figure  2.1 


about  the  professors  by  having  the  students  complete  an 
interactive  computer  survey  at  a  Naval  Postgraduate  School 
computer  terminal  with  display  screen.  Although  the  data 
collection  in  this  manner  was  not  entirely  free  from  its  own 
brand  of  difficulties,  it  eliminated  the  requirement  for  any 
paperwork  to  be  handed  out  or  returned.  All  the  results 
were  automatically  sent  to  the  researchers  computer  storage 
at  the  completion  of  the  survey.  Also,  the  data  was  in  a 
ready-to-use  state  by  being  contained  on  computer  disk,  and 
hence  the  need  to  transcribe  results  disappeared. 

Unfortunately,  there  are  numerous  contingencies  for 
which  a  computer  programmer  must  plan  in  order  to  design  a 
computer  survey  that  is  simple,  thorough,  and  user-friendly. 
The  interactive  computer  survey  designed  for  this  study 
served  its  purpose  well.  An  example  of  this  program, 
written  in  foctran,  is  included  in  Appendix  D. 

B.  THE  COMPUTES  SOHVET 

Several  important  requirements  that  had  to  be  addressed 
in  designing  the  interactive  survey  included  simplicity, 
thoroughness  and  brevity.  It  was  considered  important  to 
collect  accurate  responses  and  to  limit  the  workload  as  much 
as  possible.  Each  judge  responded  to  one  hundred-seventy 
items  on  the  average.  Even  with  this  many  questions  to 
answer,  each  student  appeared  to  have  completed  the  survey 
on  the  order  of  twenty  minutes.  Even  still,  more  work 
should  be  done  to  reduce  the  amount  of  information  asked  of 
each  judge. 

The  nature  of  the  computer  survey  progressed  as  follows. 
Each  judge  was  required  to  link  to  the  researchers  computer 
disk,  and  once  accessed,  issue  a  command  to  run  the  execu¬ 
tive  program  that  drove  the  survey.  Immediately,  a  panel  of 
the  names  of  the  professors  in  the  Operations  Research 
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III.  METHODOLOGY 


A.  LAYISG  THE  FOUNDATION 

The  first  step  in  this  study  to  understand  perceptions 
was  to  decide  upon  a  control  group  of  students  from  whom 
data  would  be  obtained.  In  this  particular  instance,  the 
judges  were  twenty- three  Operations  Research  students  from 
the  section  that  graduates  in  March  of  1985.  The  twenty- 
three  students  included  two  foreign  nationals,  two  Marine 


Corps  officers. 

four 

women  naval 

officers  and 

fifteen 

male 

naval  officers 

all 

with  various 

educational 

and  career 

back  grounds. 

The  information 

that 

was  to 

be  gleaned 

from 

these 

students  was 

simply 

this: 

How 

do  professors  in 

the 

Operations  Research  department  at  the  Naval  Postgraduate 
School  differ,  and  in  what  ways  are  they  similar  in  teaching 
styles  and  methods?  There  were  several  methods  available  to 
obtain  this  information.  Most  simply,  a  hand  written  survey 
with  many  questions  c  ^ld  have  been  developed  and  handed  to 
each  student  to  complete  and  return.  The  decision  was  made 
to  handle  the  data  collection  via  a  computer  interactive 
survey  in  order  to  minimize  the  logistics  and  data  manipula¬ 
tion  problems  associated  with  a  hand  written  survey.  The 
logistics  problem  was  not  a  great  problem  per  se,  it  just 
meant  that  a  hand  written  survey  had  to  be  distributed  and 
collected  with  plenty  of  opportunity  for  the  surveys  to 
become  misplaced.  From  a  data  manipulation  point  of  view, 
the  data  collected  from  a  hand  written  survey  would  almost 
certainly  have  to  be  recorded  on  computer  in  order  to  be 
able  to  use  the  results  easily  and  guickiy.  As  a  result, 
the  decision  was  made  to  attempt  to  collect  the  information 
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[Bef.  6;  p.  38].  The  three  types  of  problems  that  result 
from  uncertainties  about  the  covariance  matrix  and  the 
underlying  causal  structure  are;  1)  specific  covariance 
matrices  can  be  created  by  factor  models  with  the  same 
number  of  common  factors  but  with  different  factor  loadings; 
2)  specific  covariance  matrices  can  be  created  by  factor 
models  with  different  number  of  common  factors;  3)  certain 
covariance  matrices  can  be  created  by  factor  analytic  causal 
models  as  well  as  non-factor  analytic  causal  models  [Ref.  6: 
p.  38], 

Two  assumptions  commonly  made  in  factor  analysis  are  the 
postulate  of  factorial  causation  and  the  postulate  of  parsi¬ 
mony.  Basically,  the  postulate  of  factorial  causation 
requires  that  the  researcher  show  that  the  originally 
observed  variables  are  a  linear  combination  of  some  causal 
variables  [Ref.  6:  p.  43].  The  postulate  of  parsimony 
allows  the  researcher  to  assume  a  factor  model  with  a 
smaller  number  of  factors,  given  that  two  factor  models  with 
different  numbers  of  factors  have  the  same  covariance 
structure  [Ref.  6;  p.  44], 

G.  CLUSTER  Mil  ISIS 

A  final  topic  used  to  assist  in  this  study  is  cluster 
analysis.  Cluster  analysis  is  a  class  of  technigues  that 
typically  place  objects  into  groups  or  clusters  suggested  by 
the  data  such  that  objects  in  a  cluster  tend  to  be  similar 
to  objects  in  the  same  cluster  and  dissimilar  to  objects  in 
other  clusters  [Ref.  7],  The  type  of  cluster  analysis 
followed  in  this  study  is  disjoint  cluster  analysis  whereby 
objects  may  belong  to  one  and  only  one  cluster  as  opposed  to 
hierarchical  cluster  analysis  where  one  cluster  might  be 
contained  within  another. 
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dependent  variable.  The  closer  the  value  of  R2  is  to  0,  the 
sore  likely  it  is  that  the  model  is  inappropriate  in 
accounting  for  the  variation  in  the  dependent  variable.  The 
reasons  for  a  low  value  of  R2  are  several.  For  one  thing, 
the  relationship  under  investigation  may  not  be  a  linear 
one.  If  this  is  the  case,  linear  regression  modelling  is  no 
longer  a  satisfactory  method  to  use  to  describe  the  rela¬ 
tionship.  However,  given  that  the  relationship  is  truly  a 
linear  relationship,  the  reason  for  a  low  value  of  R2  could 
be  the  result  of  specification  error.  Hhat  is  meant  by  this 
is  that  the  dependent  variable  for  which  an  explanation  is 
sought,  is  being  explained  by  an  inappropriate  set  or  an 
insufficient  number  of  independent  variables. 

The  artwork  in  regression  modelling  as  well  as  in  alter¬ 
native  statistical  modelling  methods  comes  from  being  able 
to  suggest  (or  divine)  the  correct  explanatory  variables. 
In  this  particular  study,  the  explanatory  variables  used 
came  from  current  student  opinion  forms  and  suggestions  from 
previous  students  in  the  Operations  Research  curriculum. 

F.  FACTOR  ANALYSIS 

As  mentioned  before,  factor  analysis  attempts  to  repre¬ 
sent  a  set  of  observed  variables  in  terms  of  a  smaller  set 
of  hypothetical  variables.  The  hypothetical  variables  are 
chosen  to  account  for  the  covariation  among  the  originally 
observed  variables.  The  number  of  common  factors  present 
among  the  observed  variables  can  be  estimated  from  the  rank 
of  the  adjusted  correlation  matrix. 

Difficulties  in  factor  analysis  arise  when  the  factor 
loadings  are  not  known  and  have  to  be  estimated  from  the 
covariance  or  correlation  matrix.  The  problem  is  that  given 
the  correlation  matrix  for  the  observed  variables  is  known, 
any  one  of  many  causal  structures  could  have  produced  it 
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It  is  not  always  possible  to  determine  if  the  local 
minimum  is  also  the  global  minimua,  but  some  techniques 
exist  to  help  verify  that  this  is  so.  For  example,  starting 
the  minimization  process  from  several  different  initial 
configurations  and  comparing  the  final  solutions  will  indi¬ 
cate  if  the  same  local  minimum  is  achieved.  The  final 
configuration  with  the  best  stress  value  is  most  likely  the 
global  minimum.  There  is  nothing  to  guarantee  that  one  will 
always  achieve  the  global  minimum,  but  this  is  a  common 
problem  typical  of  non-linear  optimization  problems.  In  any 
case,  the  configuration  is  only  useful  if  in  the  end  it 
makes  sense  and  gives  insight  to  the  experimenter  [Hef.  4: 
p.  119]. 

E.  HD1TIPLE  LIBEAB  BEGHESSIOH 

Once  the  mul tid imesional  scaling  algorithm  computes  the 
configuration  with  the  lowest  stress,  the  researcher  would 
like  to  determine  the  specific  dimensions  that  underlie  the 
data  structure.  One  way  to  do  this  is  to  assume  that  a 
linear  relationship  exists  between  a  dependent  variable  and 
several  independent  variables.  Many  linear  regression 
models  assume  that  the  proportion  of  explained  variation  of 
the  dependent  variable  is  the  sum  of  additive  effects  of 
statistically  significant  independent  variables  [Hef.  5:  p. 
54],  Other  regression  models  allow  for  interactive  effects 
between  independent  variables.  The  dependent  variable  is 
said  to  be  regressed  over  the  independent  variables.  The 
result  of  this  regression  process  is  the  coefficient  of 
multiple  determination,  R 2  .  The  value  of  R2  indicates  the 
amount  of  variation  in  the  dependent  variable  explained  by 
the  independent  variables.  The  value  of  R2  ranges  between 
0  and  1.  The  higher  the  value  of  R*  ,i.e.,  the  closer  it  is 
to  1,  the  better  the  model  explains  or  predicts  the 
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D.  THE  HOLTID IHEHSI 0 HAL  SCALING  ALGOBITHH 


Since  the  mathematical  technique  for  determining  the 
optimal  configuration,  and  therefore  optimal  stress,  is 
somewhat  complicated,  only  a  brief  description  of  what  is 
considered  necessary  will  be  described  here.  Suppose  that  t 
dimensions  are  selected  to  describe  a  configuration  of  n 
points.  Then 

(r„  ,...xlt  ,  ...xn;  ,  •••x.,t) 

can  be  used  to  describe  a  particular  configuration  in  multi¬ 
dimensional  space.  For  this  particular  configuration,  a 
specific  value  of  stress  exists.  The  overall  objective  is 
to  make  the  stress  value  as  small  as  possible.  This  turns 
out  to  be  a  minimization  problem  of  multiple  variables  and 
is  handled  by  the  method  of  steepest  descent.  Specifically, 
the  algorithm  begins  at  an  arbitrary  configuration  and 
attempts  to  improve  itself  by  moving  in  the  direction  that 
improves  or  minimizes  the  stress  value  quickest.  The  direc¬ 
tion  of  movement  is  known  as  the  negative  gradient  and  can 
be  evaluated  from  the  partial  derivarives  of  the  function 

S=£(xi(  ,xa  ,...xflt) 

[Bef.  4;  p.  118].  For  example, 

(—  ^S/  x„  ^ S/  X|t  ,.  .  .,  — i S/  Xjh  ) 

is  the  negative  gradient.  Once  the  configuration  reaches 
the  point  at  which  it  can  no  longer  improve  in  a  particular 
direction,  the  new  negative  gradient  is  calculated  and  the 
process  continues.  Finally,  when  a  configuration  can  no 
longer  proceed  in  any  direction  with  improvement,  it  has 
reached  a  local  minimum.  Hopefully,  the  local  minimum  is 
also  the  global  minimum,  but  this  is  not  necessarily  true 
all  the  time. 
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are  methods,  however,  which  help  to  indicate  why  one  choice 
of  dimensionality  would  be  more  appropriate  than  others. 
The  most  obvious  of  these  is  to  compare  how  stress,  the 
goodness-of-fit  measure,  improves  as  a  function  of  dimen¬ 
sion.  One  way  to  do  this  is  to  compute  the  best  configura¬ 
tion  for  several  dimensions  and  create  a  plot  of  stress  vs. 
dimension  to  visually  compare  the  results.  It  should  be 
pointed  out  that  the  more  dimensions  one  uses  to  explain  or 
interpret  the  data,  the  lower  the  stress  becomes.  When  the 
number  of  dimensions,  t,  exceeds  the  number  of  objects  minus 
one,  n-1,  the  stress  will  always  be  zero  [Bef.  2:  p.  16]. 
What  one  wants  to  look  for  is  that  dimension  above  which  the 
stress  improves  only  slightly.  If  the  data  is  good,  a 
noticeable  elbow  will  show  up  in  the  plot  to  indicate  the 
appropriate  dimension.  Be  were  not  so  fortunate  in  our 
present  study.  Figure  2.3  illustrates  a  stress  versus 
dimension  plot. 

Probably  more  than  anything  else,  interpretability 
should  be  considered  a  key  criterion  to  use  in  selecting  the 
appropriate  dimension  for  analysis.  If  it  is  possible  to 
interpret  the  results  of  an  MDS  configuration  in  two  dimen¬ 
sions  more  readily  than  in  say  three  dimensions,  even  though 
the  stress  is  lower  in  three  dimensions,  one  should  consider 
employing  the  two  dimensional  interpretation.  A  final 
criterion  suggested  by  Kruskal  [Bef.  2:  p.  16],  depends  upon 
the  accuracy  of  the  data.  If  an  independent  estimate  exists 
to  corroborate  error  free  or  near  error  free  data,  then  one 
is  allowed  to  extract  more  dimensions  than  one  would  under 
more  error  prone  conditions.  When  all  is  said  and  done,  the 
choice  of  dimensionality  rests  largely  upon  the  experience 
of  the  experimenter. 
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Piqure  2.2  The  Degree  to  vhich  the  Data  do 
not  Satisy  the  hoaotoaicity  Hegaxrenent. 


Table  of  Stress 

TABLE  1 

Valves  According  to  Kruskal 

STRESS  VALUE 

GOODNESS  OF  FIT 

.20 

300R 

.  1  0 

FAIR 

.05 

GO00 

.0  25 

Excellent 

0.3 

perfect 

Figure  2.2  illustrates  an  example  of  what  we  mean  by  a 
non- perfect  match  between  the  dissimilarity  and  the  distance 
data.  Here,  the  deviations  measured  along  the  horizontal 
distance  axis  ,d-^  ,  between  the  starred  coordinates  and  the 
circular  coordinates,  indicate  the  degree  to  which  this 
configuration  does  not  meet  the  monotonicity  requirement. 

A 

The  values  d;;  are  defined  to  be  those  numbers  measured  from 
the  horizontal  X  axis  that  minimize  stress  subject  to  the 
constraint  of  monotonicity  [Hef.  2:  pp.  8-9]. 

C.  DIHEISIOHAl  HEQOIBEHBBTS 

The  choice  of  how  many  dimensions  are  required  to 
completely  specify  the  output  from  the  multidimensional 
scaling  program  is  certainly  not  intuitively  obvious.  There 
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in  greater  than  three  dimensions  becomes  extremely  difficult 
for  anyone  trying  to  discover  the  meaning  of  the  groupings. 
The  stress,  or  goodness-of-f it  measure,  which  is  described 
in  the  next  section,  is  usually  expected  to  reveal  the  best 
dimension  for  analysis. 

6.  GOODHESS  OF  FIT: THE  STRESS 

A  performance  indicator  for  each  choice  of  t  is  needed. 
The  customarily  used  function  is  known  as  the  stress. 
Stress  incorporates  a  fitting  technigue  that  measures  the 
degree  of  nonmonotonicity  between  the  dissimilarities  4V,  , 
and  the  distances  d%j  .  If  a  configuration  of  points 
existed  such  that  a  perfect  monotone  relationship  prevailed 
between  the  dissimilarity  data  and  the  distance  data,  then  a 
perfect  match  would  occur  and  the  stress  would  be  zero  for 
that  particular  dimension  size  and  all  dimensions  greater. 
Unfortunately,  zero  stress  rarely  if  ever  occurs  naturally 
in  data  samples.  The  best  choice  then  is  to  determine  that 
configuration  of  points  which  minimizes  the  stress  for  each 
choice  of  dimension,  t.  The  method  used  to  determine  stress 
uses  least  squares  monotonic  regression,  suitably  normalized 
to  produce  a  non-dimensional  value  that  indicates  goodness 
of  fit  [Ref.  2:  pp.  2-3].  The  lower  the  value  of  stress, 
the  tetter  the  fit.  Kruskal  [Ref.  2:  p.  3],  has  been  able 
to  associate  a  verbal  description  with  some  specific  values 
of  stress  based  upon  his  experience.  These  values  are  indi¬ 
cated  in  Table  1  .  In  almost  every  instance,  by  increasing 
the  number  of  dimensions  that  describe  the  data,  the  value 
of  the  stress  decreases.  However,  one  generally  anticipates 
that  the  amount  of  benefit  associated  with  incrementing  the 
dimensionality  is  marginally  insignificant.  In  other  words, 
a  tradeoff  exists  between  keeping  the  output  in  a  lower 
dimension  to  make  interpretation  easier,  if  the  stress 
improves  only  slightly. 
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In  this  study  the  dissimilarity  values  and  were 

assumed  to  be  the  same  (symmetric  matrix),  and  the  values 
,  were  ignored.  This  results  in  a  partial  matrix  with  upper 
triangular  or  lover  triangular  form  not  including  the  aain 
diagonal.  The  total  possible  pairwise  comparisons  are  then 
n(n-1)/2. 

The  actual  distance  between  professors  i  and  j,  denoted 
d”  ,  is  calculated  as  a  euclidian  distance  in  the  following 
manner  for  t  dimensions 


‘■j "  VI;> 


)*♦. 


like  the  similarity/dissimilarity  matrix,  the  end  result  is 
a  matrix  of  distances  such  that  d;  =  d  ;  and*d-  *0.  He  point 
out  here  that  although  we  have  specified  the  euclidean 
distance  as  the  method  by  which  distances  are  computed  in 
the  computer  algorithm,  it  is  possible  to  substitute  non* 
euclidean  distances  of  the  form 


a  <*#y) 


for  r>l. 


In  the  field  of  mathematics,  these  distances  are  known  as 
Hinkowski  r-metrics  and  are  true  distances  in  the  sense  that 
they  satisfy  the  triangle  inequality 


dr(x,z)<  dr  (x,  y)  +dr  (y,  z)  . 

Euclidean  distances  and  Hinkowski  r-mettics  share  many  prop* 
erties,  however,  they  do  differ  when  it  comes  to  rotating 
the  solution.  When  rotations  are  involved,  any  rigid  rota* 
tion  ..eaves  euclidean  distances  unchanged.  The  only  rigid 
rotations  that  leave  non*euclidean  distances  unchanged  are 
rotations  that  transform  all  permutations  of  coordinate  axes 
into  coordinate  axes  [Bef.  2:  pp.  22*23]. 

Finally,  the  number  of  dimensions  used  to  output  the 
final  mapping  is  not  restricted  in  any  way  mathematically 
other  than  HT<N{N-1)/2.  However,  any  visual  interpretation 
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judges.  Each  professor  can  be  thought  of  as  a  point  in 
multidimensional  space.  If  it  takes  t  dimensions  to  accu¬ 
rately  describe  this  aultidiaensional  space,  then  the  coor¬ 
dinate  describing  professor  x  would  be 

For  n  professors  the  coordinate  system  that  results  looks 


like 


x,  »  (X,  ****X\s  *•*  ) 


*r  =(*r, 


'• * ,Xrk J 


*  *X.-rtL  ) 


Thus,  the  entire  space  contains  a  configuration  of  n  points 
each  of  t  components. 

Each  judge  or  student  is ' asked  to  complete  a  survey  that 
requires  him  to  provide  a  value  corresponding  to  how  similar 
or  dissimilar  he  perceives  each  pairwise  combination  of 
professors  to  be.  The  scale  used  in  the  survey  associates 
the  value  1  with  the  meaning  very  similar  and  the  value  of  9 
with  the  meaning  very  dissimilar.  In  order  to  determine 
rather  than  impose  the  relationships  between  professors,  the 
characteristics  on  which  the  professors  are  scored  are  not 
specified  at  this  time  [Ref.  3:  p.  9].  The  notation  for 

dissimilarity  data  in  this  study  is  <J\j  .  This  value  repre¬ 
sents  the  perceived  dissimilarity  be' ween  professor  i  and 
professor  j.  The  end  result  of  the  data  collection  is  a 
matrix  of  dissimilarity  values  for  n  stimuli  that  looks  like 
the  following, 

f  6  J 


r 


It  can  be  seen  that  an  ascending  pattern  is  created  between 
the  distances  and  the  dissimilarities.  One  night  even 
suggest  that  the  relationship  is  linear,  and  could  be 
described  by  an  equation  of  the  form  a  +  bx. 
Multidimensional  scaling  methods  that  use  a  formula  to 
describe  distance  as  a  function  of  dissimilariti  es  are  known 
as  metric  MDS.  Metric  MDS  uses  the  numerical  properties  of 
the  proximity  data  to  determine  distances.  Another  means  by 
which  distances  are  created  from  dissimilarity  data  without 
using  the  numerical  properties  of  the  dissimilarity  data  is 
known  as  nonmetric  MDS.  Honmetric  MDS  relies  totally  upon 
the  rank  ordering  of  the  dissimilarities  to  produce  the 
distance  data  [Bef.  3:  p.  22].  One  would  normally  expect 
small  dissimilarities  to  correspond  to  small  distances  and 
large  dissimilarities  to  correspond  to  large  distances. 
Although  this  relationship  is  rarely  a  perfect  one,  the  rank 
orderings  of  the  dissimilarity  data  is  usually  enough  to 
create  a  good  fit.  Shepard  and  Kruskal  [Ref.  2:  p.  2],  have 
done  a  substantial  amount  of  work  in  nonmetric  MDS  and  this 
particular  method  will  be  followed  in  this  study. 

A  review  of  some  notation  might  be  appropriate  at  this 
point.  The  n  objects  about  which  the  investigator  is  trying 
to  ascertain  some  fundamental  relationship,  can  really  be 
any  set  of  stimuli.  For  example,  one  might  be  interested  in 
discerning  the  perceived  distances  between  political  candi¬ 
dates  with  hopes  of  discovering  what  issues  or  dimensions 
really  set  them  apart  from  each  other  in  the  minds  of  the 
judges.  Or,  one  might  be  interested  in  discovering  the 
perceived  distances  between  countries  in  order  to  explain 
how  some  countries  might  react  in  some  political  or  economic 
situations.  For  this  study,  the  n  objects  are  the 
professors  in  the  Operations  Pesearch  department  and  we  are 
interested  in  determining  what  factors  cause  the  professors 
to  be  similar  or  dissimilar  in  the  minds  of  a  fixed  set  of 
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department  would  appear,  and  the  judge  would  be  asked  to 
indicate  those  professors  from  whom  he  had  taken  a  course. 
After  selecting  his  own  subset  of  professors,  the  judge  was 
asked  to  rate  each  pairwise  combination  of  professors  in 
terms  of  similarity  or  dissimilarity  of  teaching  style.  If 
the  judge  had  observed  n  professors,  this  meant  that  a  judge 
would  have  to  respond  to  at  least  n(n-1)/2  prompts  for  this 
proximity  data.  For  this  study,  the  sixteen  professors  are 
identified  by  using  the  letters  A  thru  P.  The  scale 
presented  to  the  judge  ranged  from  a  value  of  1  meaning  very 
similar  to  a  value  of  9  meaning  very  dissimilar.  The  judges 
were  not  limited  to  integer  responses,  but  real  responses 
were  restricted  to  one  decimal  place.  It  was  assumed  that 
the  proximity  scale  was  an  interval  scale  meaning  the 
distance  in  similarity  or  dissimilarity  values  between  say 
values  2  and  3  was  equal  to  the  distance  in  dissimilarity 
between  values  8  and  9. 

Once  all  the  proximity  data  had  been  collected,  the 
judges  were  then  asked  to  score  the  professors  with  respect 
to  several  bipolar  scales.  In  the  case  where  students  had 
taken  a  professor  for  more  than  one  course,  the  student  was 
instructed  to  respond  to  the  bipolar  scales  based  on  the 
last  course  taught  by  the  professor.  The  scales  used  in  the 
survey  appear  in  Table  2  and  were  suggested  by  previous 
Operations  Research  students.  The  bipolar  scales  also 
ranged  in  value  from  1  to  9.  Once  each  judge  completed  the 
survey,  his  or  her  results  were  automatically  sent  to  the 
researchers  computer  disk  for  subsequent  evaluation. 

In  addition  to  the  data  collected  from  the  computer 
interactive  survey,  information  gathered  from  student 
opinion  forms  (SOF's)  were  used  to  try  to  interpret  charac¬ 
teristics  of  the  different  professors.  Only  those  student 
opinion  forms  from  those  classes  taught  by  the  professors 
selected  in  the  interactive  survey  were  considered.  The 
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bipolar  scales  used  is  current  SOF  forms  appear  in  Table  3  . 
One  problem  encountered  in  the  evaluation  of  these  forms 
dealt  with  maintaining  the  purity  of  the  control  group.  For 
the  most  part,  the  SOF's  were  completed  by  the  twenty-three 
students  in  the  control  group.  However,  there  were  some 
instances  where  other  students  either  from  other  Operations 
Eesearcl  sections  or  from  other  curriculums  were  included  in 
the  evaluation  process.  The  information  collected  from  the 
SOF's  represented  different  scales  upon  which  to  measure 
teacher  performance.  The  SOF  data  was  originally  converted 
to  an  interval  scale  on  a  range  different  from  the  scales 
used  in  the  interactive  survey.  Because  linear 
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TABLE  3 

Bipolar  Scales  Used  in  the  SOF  Forns 


l 

1.  course  organization 

2.  TIME  IN  CLASS  SPENT  EFFECTIVELY 

3.  INSTRUCTOR  KNOWS  WHEN  STUDENTS  DONT • T  UNOEHSTANO 
MATER  I  AC 

а.  o t ff i cult  concepts  maoe  understandable 

5.  CONFIDENCE  IN  INSTRUCTORS  KNOWLEDGE  IN  SUBJECT 

б.  FELT  free  TO  ASK  QUESTIONS 

7.  INSTRUCTOR  PREPARED  FOR  CLASS 

8.  INSTRUCTORS  OBJECTIVES  MAOE  CLEAR 

<J.  INSTRUCTOR  WADE  COURSE  WORTHWHILE  LEARNING  EXPERIENCE 

10.  INSTRUCTOR  STIMULATED  INTEREST  IN  SUEjECT  AR£A 

11.  INSTRUCTOR  CARED  AEOUT  STUOENT  PROGRESS  AND  010  HIS 
SHARE  IN  HELPING  TG  LEARN 


tr  a  ns  forma  tions  of  the  fora  a+hx  are  allowed  on  interval 
scales,  the  scale  used  for  the  SOF’s  was  changed  to  match 
the  scale  used  in  the  interactive  survey.  In  the  case  of 
each  set  of  scales,  a  separate  regression  analysis  and 
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factor  analysis  vas  done.  One  key  parameter  we  were 
interested  in  measuring  was  the  correlation  between  how  the 
judges  evaluated  the  professors  overall  performance  at 
different  times  in  the  curriculum.  SOF  data  was  only  avail* 
able  through  the  end  of  the  summer  quarter  of  1984  (the  end 
of  the  judge's  sixth  quarter) .  Since  the  judges  completed 
the  computer  survey  at  the  beginning  of  the  eigth  quarter, 
only  those  professors  taken  through  the  end  of  the  sixth 
quarter  were  evaluated.  This  meant  that  a  few  professors 
were  not  included  in  the  final  evaluation. 

C.  OUTPUT  FEOU  HDS 

The  proximity  or  dissimilarity  data  obtained  from  the 
interactive  survey  was  used  as  a  direct  input  to  a  multidi¬ 
mensional  scaling  algorithm  KYST,  partially  developed  by 
Kruskal  [Bef.  1:  p.  1].  The  proximity  input  values  for  this 
study  appear  in  Appendix  A.  The  output  from  the  multidimen¬ 
sional  scaling  program  includes  visual  plots  or  spatial  maps 
depicting  the  professors  positions  in  multidimensional  space 
projected  down  to  two  dimensions  for  visual  display.  Also 
included  as  output  are  coordinates  for  each  professor  in 
multidimensional  space.  The  task  that  remained  was  to 
define  the  variables  that  best  explained  the  location  of 
each  professor. 

One  method  available  for  determining  what  characteris¬ 
tics  explain  the  orientation  of  the  professors  from  the 
multidimensional  scaling  output  is  multiple  linear  regres¬ 
sion.  The  median  value  from  each  bipolar  scale  is  regressed 
over  the  coordinate  positions  of  each  professor.  From  the 
stress  versus  dimension  curve  it  was  decided  to  concentrate 
on  a  four  dimensional  interpretation,  although  three  and 
five  dimensional  interpretations  were  considered  also.  The 
values  of  stress,  the  goodness-of-f it  function,  turned  out 
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to  be  0.249  for  four  dimensions  and  0.293  for  three 
dimensions.  Neither  of  these  values  indicate  a  very  good 
fit  according  to  Kruskals*  own  personal  experience.  This 
large  value  of  stress  was  an  early  indication  that  this 
linear  model  might  not  be  appropriate  in  explaining  teacher 
performance. 

Figure  3.1  shows  the  resulting  regression  weights  or 
direction  cosines  corresponding  to  each  multiple  correlation 
for  the  four  dimensional  solution.  The  direction  cosines 
are  regression  weights  normalized  so  that  their  sum  of 
sguares  eguals  1.0  for  every  scale  [Ref.  3:  p.  37].  For 
example,  when  regression  weights  of  0.4178,  0.8959,  -0.1508, 
an<!  -0.0026  are  given  to  dimensions  1,2,3  and  4  respec¬ 
tively,  the  multiple  correlation  between  the  resulting  coor¬ 
dinate  positions  and  the  respective  bipolar  scale  is  0.581. 

A  bipolar  scale  will  provide  a  good  interpretation  of  a 
dimension  when  its  multiple  correlation  coefficient  is  high. 
A  value  above  .90  is  desired.  The  values  achieved  in  this 
study  were  low.  7alues  of  R*  close  to  0.5  were  typical. 
Also  a  requirement  for  good  dimensional  interpretation  is  a 
high  regression  weight  on  the  dimension  it  most  nearly 
explains.  The  results  obtained  from  the  bipolar  scales  used 
in  this  study  are  examined  in  the  next  chapter. 

D.  OUTPUT  FBOH  FACTOR  ANALYSIS 

Having  looked  at  regression  modelling  as  an  approach  to 
interpreting  dimensions,  factor  analysis  was  also  considered 
a  possible  means  of  identifying  linear  factors  that  would 
help  describe  teacher  performance  from  the  data  sample.  The 
statistical  analysis  package,  SAS,  was  used  to  generate  two 
separate  factor  analysis  outputs.  One  factor  analysis  was 
completed  for  the  bipolar  scales  associated  with  the 
computer  survey  and  another  was  done  on  the  additional 
bipolar  scales  from  the  SOF  forms. 
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positive  poles  of  normalized  regression  coefficients 

°*T1nG  SCALES  (DIRECTION  COSINES*  multiple 


CQNRU  T£R  BI POLAR  SCALES 

DIM  1 

01*2 

0 1»3 

0f*4 

conoei,  at  iom 
C0E*P  ICI6NT 

t  . 

CLASS  SIZE 

0.395* 

0.0 190 

0.9172 

-0. 0*34 

•  262 

2  • 

theoretical  vs. 
applieo 

0.2603 

•0.8367 

-0.4816 

-0.0096 

•  5  10 

3. 

GRADING  POLICY 

0.31 97 

•0.6784 

-0.3689 

•0.  54  89 

•  4  14 

4. 

RACE  OF  COURSE 

0.2327 

•0.5951 

-0.4877 

-0.5865 

.453 

5. 

EFFORT  REQUIRED 

OUTSIDE  OF  CLASS 

0.  i  391 

•0.9  142 

-0.3253 

-0. 1975 

.3  76 

6. 

C  OURSE  PEL  1 EO-UPON 
PREREQUISITES 

0 .0696 

•0.3655 

-0.5769 

-0. 72  70 

.501 

SCF 

B I  POLAR  SCALES 

• 

7. 

COURSE  ORGANIZATION 

0.59*2 

0 .6474 

-0.4223 

0.2219 

•  502 

e. 

TI»E  IN  CLASS  SPENT 
EFFECTIVELY 

0.643$ 

0.6  1  t  a 

-0.0025 

0.4597 

.493 

9. 

INSTRUCTOR  KNOWS  .MEN 
STUOENTS  OONT  UNDER¬ 
STAND  material 

0.9337 

0.330  1 

0.01  1  3 

0. 1375 

•  285 

10. 

DIFFICULT  CONCEPTS 

MAOE  UNDERSTANDABLE 

0.5412 

0.1880 

-0.461  2 

0.2103 

•  444 

1  1  • 

CONFIDENCE  in  instruc¬ 
tors  knowledge  in  sub¬ 
ject 

0.2928 

0.1 832 

0.6859 

-0.6404 

•  095 

f  2. 

“ELT  FREE  TO  ASK 
questions 

0. 4872 

•0.0Q87 

0.0466 

-0.1238 

•  405 

1  3. 

INSTRUCTOR  PREPARED 

FOR  CLASS 

0.7263 

0.5747 

-C.3012 

0.2265 

.447 

|4. 

instructors  objectives 
naoe  clear 

0.41 78 

0.8959 

-0.1508 

-0.0026 

.561 

1  5  ■ 

instructor  MAOE  COURSE 
worthwhile  learning 
experience 

0.706$ 

0.4218 

-0.4327 

-0.3684 

.4  78 

I  6  . 

INSTRUCTOR  STIMULATED 
INTEREST  in  subject 
AREA 

0.6352 

0.2174 

-0.6073 

-0.4245 

.296 

17. 

INSTRUCTOR  CARED  ABOUT 
STUOENT  PROGRESS  aNC 

0.5310 

0.2991 

-0.4 143 

-0.2193 

.532 

OIO  His  SHAPE  IN  HELP¬ 
ING  TO  L*ARN 


Figure  3.1  Eegression  Heights  for  the 
Four  Diaensional  Solution. 
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Initially,  a  correlation  matrix  was  produced  from  the 
raw  input  data.  Additionally,  the  common  factors  and  the 
factor  loadings  for  each  observed  variable  or  bipolar  scale 
was  produced.  Orthogonal  rotations  were  effected  to  produce 
simple  structure.  A  total  of  two  common  factors  were 
created  from  the  bipolar  scales  of  the  interactive  survey, 
and  one  common  factor  was  produced  from  the  SOF  data.  The 
specifics  of  the  factor  analysis  output  are  discussed  in  the 
next  chapter. 

E.  OUTPUT  FROM  CLUSTER  ANALYSIS 

As  a  final  measure,  a  disjoint  fluster  analysis  was 
performed  on  the  MDS  data.  The  clustering  routine, 
Fastclus,  was  available  from  the  statistical  analysis 
package,  SAS.  The  number  of  clusters  into  which  the  grcup 
of  instructors  were  subdivided  was  specified  by  the 
researcher  to  range  from  2  to  6.  Membership  in  a  particular 
cluster  was  determined  based  upon  the  distance  from  each 
professors  position  to  the  mean  value  of  the  cluster.  The 
output  from  the  cluster  analysis  included  identification  of 
the  cluster  to  which  each  professor  belonged. 
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17.  DATA  ANALYSIS 


A.  IITEHT  OF  THE  ASAIYSIS 

The  scope  of  this  chapter  will  be  to  analyze  the  results 
obtained  froa  the  completed  interactive  survey  in  the 
context  of  multiple  linear  regression  and  stepwise  regres¬ 
sion,  factor  analysis  and  cluster  analysis.  Additionally, 
information  gathered  from  student  opinion  forms  (SOFs)  ,  will 
be  evaluated  in  so  far  as  what  characteristics  or  bipolar 
scales  appear  to  have  been  most  important  in  describing 
students  perceptions  of  teacher  performance. 

An  issue  that  requires  explanation  before  the  analysis 
begins  concerns  a  vital  assumption  made  dealing  with  the 
scale  of  the  data.  Specifically,  can  an  arbitrarily  chosen 
numerical  scale  with  fixed  upper  and  lover  bounds,  enable 
correct  statistical  inference  from  the  data  sample?  In  our 
study,  we  provided  the  judges  with  a  numerical  scale  ranging 
in  value  from  1  to  9.  The  judges  were  allowed  to  rate  each 
professor  in  every  category  with  respect  to  this  scale.  In 
essence  we  are  imposing  an  interval  scale,  with  equally 
spaced  intervals.  let's  consider  a  judge's  response  to  the 
question  of  grading  policy.  It  may  be  that  the  judges  can 
rate  the  different  objects,  in  our  case  professors,  at  best 
on  an  ordinal  scale.  The  judge  may  be  able  to  say  that 
professor  A  is  a  harder  grader  than  is  professor  B,  but  not 
how  much  harder.  We  are  assuming  that  the  judges,  when 
responding  to  these  bipolar  scales,  realize  that  we  are 
infering  an  interval  scale  on  their  responses.  Fe  assume 
that  when  they  complete  the  survey,  that  they  realize  that 
each  integer  value  on  the  scale  divides  the  scale  into  equal 
intervals.  We  assume  that  a  judge  will  rate  each  professor 


33 


'  r 


I 


I 


knowing  that  the  distance  between  say  a  score  of  3  and  4  is 
egual  to  the  distance  between  a  score  of  5  and  6.  He  find 
ourselves  doing  this  in  order  to  follow  the  example  of 
Xruskal  as  closely  as  possible  and  because  the  statistical 
methods  used  to  evaluate  the  data  require  at  least  an 
interval  scale.  Given  that  we  are  assuming  an  interval 
scale,  we  feel  comfortable  in  transforming  the  data  with  any 
linear  transformation  of  the  form  a  ♦  bx.  The  major  point 
being  made  here  is  that  the  researcher  should  keep  in  mind 
the  scale  of  the  data  when  interpreting  the  significance  of 
the  output. 

B.  WHAT  DID  HE  LEAR1  FEOH  EDS? 

As  mentioned  in  Chapter  II,  multidimensional  scaling 
attempts  to  determine  the  structural  relationships  between  r. 
objects  from  a  matrix  or  half  matrix  of  proximity  data.  A 
major  result  of  the  process  is  the  spatial,  mapping  of  the  n 
objects,  usually  projected  down  to  the  planes  of  each  pair 
of  dimensions  for  easy  visual  interpretation.  The  final 
configuration  of  points  represents  the  best  fit  of  the  n 
objects  according  to  the  stress  criterion.  Table  4  contains 
the  final  configurations  for  all  sixteen  professors  in  four 
dimensions. 

In  order  to  determine  which  dimensionality  scheme  best 
suggests  the  characteristics  that  set  the  professors  apart 
from  one  another,  it  helps  to  review  the  spatial  maps 
created  from  the  data.  Appendices  3  and  C  contain  the 
spatial  plots  of  the  sixteen  professors  for  four  and  five 
dimensional  solutions.  Each  pairwise  combination  of  axes  is 
plotted  against  each  other.  This  spatial  orientation  can 
sometimes  suggest  from  mere  inspection  those  characteristics 
that  cause  some  professors  to  be  more  alike  than  others. 
However,  we  do  not  usually  rely  upon  visual  inspection 
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TABLE  4 


Final  Configuration  for  the  16  Professors 
in  4  Dimensions 


PROF 

55  SOI 

DIP.  1 

DIM.  2 

DIM.  3 

DIM.  4 

1. 

A 

-0.737 

0.2  49 

0.482 

0.15  0 

2. 

S 

1.2C9 

0.069 

-0.517 

0.172 

3. 

c 

-0.359 

-0.207 

-0.430 

0.094 

4. 

0 

-0.643 

-0.382 

-0.053 

0.666 

5. 

5 

-0.724 

— 0. 5  o9 

0.314 

-0.113 

6. 

F 

0  .635 

0.267 

0.042 

-0.056 

7. 

G 

-0.706 

-0.085 

-0. 154 

-0.416 

a. 

H 

0.450 

-0 . 574 

0.265 

-0.279 

9. 

T 

-0.310 

0.642 

-0.006 

0.434 

10. 

J 

-0.252 

0.364 

0.312 

-0.336 

11. 

K 

1.328 

-0.205 

0.197 

-0.030 

12. 

L 

—  C  .923 

-0.223 

-0.311 

-0.370 

13. 

M 

0.773 

0.391 

0.775 

-0.113 

14. 

N 

-0.047 

-0.254 

0.477 

0.381 

15. 

0 

-C.  155 

1 . 203 

-0.611 

-0.104 

16. 

P 

0.517 

-0.693 

-0.743 

-0.030 
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alone.  Very  often  a  plot  indicating  how  stress  improves  as 
a  function  of  dimension  gives  the  dimensional  interpreta¬ 
tion.  Figure  4.1  is  such  a  plot  for  the  data  collected  in 
this  study.  It  can  be  seen  that  a  one  dimensional  interpre¬ 
tation  yields  a  very  high  stress  value  of  0.463.  This  would 
suggest  that  a  one  dimensional  interpretation  alone  would  be 
inappropriate.  He  would  like  a  noticeable  elbow  to  occur  in 
the  stress  vs.  dimension  plot,  for  this  normally  indicates 
the  most  suitable  level  of  interpretation.  He  observe  that 
this  plot  does  not  exhibit  the  noticeable  elbow.  Instead, 
the  slope  of  the  curve  decreases  gradually  and  we  are  left 
to  look  for  other  means  to  help  determine  dimensionality. 

Kruskal  and  Wish  offer  an  alternate  method  or  rule  of 
thumb  in  choosing  dimensionality  [Ref.  3  p.  34  ].  They 
suggest  that  the  number  of  stimulus  objects  minus  one,  in 
our  case  fifteen,  should  exceed  four  times  the  dimension 
chosen  for  interpretation.  This  would  offer  a  choice  of 
dimension  no  greater  than  3.75.  They  caveat  this  statement 
by  saying  that  this  rule  has  only  been  found  to  hold  for 
three  dimensions,  and  that  further  study  is  needed  to  see  if 
it  is  appropriate  for  higher  dimensions  as  well.  He  chose 
to  emphasize  a  four  dimensional  interpretation,  however  a 
three  and  five  dimensional  interpretation  were  considered 
also. 

C.  THE  USE  OF  REGRESSION 

Once  the  choice  of  dimensionality  had  been  made,  the 
next  task  was  to  determine  which  characteristics  represented 
those  dimensions  best.  To  do  this,  we  decided,  as  do 
Kruskal  and  others  [fief.  3:  pp.  35-36],  to  use  multiple 
linear  regression  as  a  means  of  clarifying  this  issue. 
Again,  we  reiterate  that  a  separate  regression  analysis  was 
conducted  for  the  scales  used  in  the  computer  survey  as  well 
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as  the  scales  obtained  from  the  SOP  forms.  Specifically,  we 
regressed  the  median  value  of  each  bipolar  scale  over  the 
coordinates  of  each  professor,  and  recorded  a  multiple 
correlation  coefficient.  If  the  value  of  the  multiple 
correlaticn  coefficient  was  high,  i.e.,  close  to  1,  then  we 
felt  inclined  to  believe  that  this  scale  was  important  in 
distinguishing  professors.  If  the  value  of  the  multiple 
correlation  coefficient  was  low,  i.e.,  close  to  0,  then  that 
particular  scale  was  not  perceived  as  being  important. 

Tables  5,  6  and  7  illustrate  the  two  sets  of  bipolar 
scales  used  in  the  study,  the  corresponding  normalized 
regression  weights  or  direction  cosines,  and  the  multiple 
correlation  coefficients  associated  with  each  scale  for 
dimensions  3,  U  and  5.  It  can  be  seen  that  for  the  most 
part,  the  multiple  correlation  coefficients  hover  near  0.5 
and  typically  increase  as  the  the  dimension  increases. 
Figure  4.2  depicts  the  changes  in  the  multiple  correlation 
coefficient  for  each  bipolar  scale  as  the  dimension  for 
interpretation  changes.  In  the  table,  bipolar  scales  1  thru 
6  refer  to  the  scales  from  the  computer  survey  and  bipolar 
scales  7  thru  17  refer  to  those  scales  from  the  SOF  form. 

Since  none  of  the  scales  exhibited  truly  high  correla¬ 
tion  coefficients,  we  chose  those  that  had  the  highest 
values  and  used  them  to  interpret  the  dimensions  in  our 
study.  The  regression  analysis  on  the  computer  survey 
scales  suggested  that  these  scales  were  inappropriate  indi¬ 
cators  for  this  set  of  judges.  For  the  four  dimensional 
analysis,  we  see  that  the  bipolar  scales  ’theoretical  vs. 
applied  course*  and  ’course  relied  upon  prereguisites ’  have 
the  highest  correlation  coefficients  for  that  group  of 
scales.  Specifically,  the  correlation  coefficients  are  .510 
and  .501  respectively.  The  scale  ’applied  vs.  theoretical 
course'  loads  heavily  on  dimension  2  with  a  normalized 
regression  weight  of  -0.8367.  Thus,  it  would  seem  that  the 
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TABLE  5 


Regression  Heights  and  Multiple  Correlation 
Coefficients  for  the  Bipolar  Scaj.es  in  3  Dimensions 


a  if  Tur^rlf CF  NORMALIZED  REGRESS  I  ON  CC=F=tC 

R4TIJG  SCAUeS  (DIRECTION  CO.'INFSl 

12  MTS 

,4ULT  I  P  L- 

c  cm  puts  p  ei  pclap  scales 

0IM1 

0172 

DIMS 

CDS  RELAT'D' 
COEFf  'Cirri' 

l.  CLASS  SI Z: 

-0.3330 

0.0124 

0.7237 

.245 

2.  THEORETICAL  VS. 
APPLIED 

-O.Z55T 

—  C. 7  464 

-0.6144 

.557 

J.  GRACING  POLICY 

-0.3794 

-C.  6005 

-C. 7039 

.313 

4.  PACE  OF  COURSE 

-0.2970 

-C. 5141 

-0. dQ4  7 

.338 

5.  EFF  CRT  RP2UIRF3 
OUTSIDE  3?  CLASS 

-0.1553 

— C. 7963 

-0. 5346 

.313 

6.  COURSE  PEL  TF^-jPCf' 
PRLREJUI  St  TES 

-0.0859 

-C.3054 

-0.9483 

.329 

SCF  8IPCLA0  SCALES 

7.  COURSE  ORGANIZATION 

-0.61  J8 

C. 7432 

-0.2731 

.4  76 

8.  TIME  IN  CLASS  SPCflT 
EFF=CT! VEl  Y 

-0. 7417 

C.6233 

0.2464 

.428 

8.  INSTRUCTOR  KNOWS  WHF' 
STUDENTS  .7 CNF  uNCfcP- 
STAMO  '<  4  T  £  R  IAL 

-0.  70  a2 

0.40  38 

0.1098 

.235 

1C.  D I  F  F ICULT  CONCEPTS 
MADE  JN  0£R  STANOA  SLE 

-0. 9633 

0.2921 

-0.4115 

.442 

11.  CONFIDENCE  IN 

IMSTRur  TORS  KNOW LEDGC 
IN  SUBJEC' 

-0.2878 

0.0078 

0.9577 

.0  76 

12.  FELT  S3  sc  7Q  ASK 

CUESTIQNS 

-0.7975 

-C. 0345 

0.0614 

.394 

13.  INSTRUCTOR  PREPARED 
FCR  CLASS 

-0.  7567 

0.63  72 

-C.1461 

.407 

1A.  INS  TF  UC  TCR  S  OBJEC¬ 
TIVES  MAOS  CLEAR 

-0.  42  <.4 

C.8983 

0.1142 

.503 

IS.  INSTRUCTOR  MAOF 

COURSE  WORTFWHILE 

L2  ARMING  EXPERIENCE 

-0. 7098 

C. 5354 

-0.3913 

.463 

la,  INSTRUCTOR  STTMULA^Fn 
INTEREST  IN  SUBJECT  ' 
AP.SA 

-0.8470 

C. 392 1 

-0.O539 

.239 

17.  INSTRUCTOR  CAREC  -0.3042 

ABOUT  57UC"NT  =PrG?=S3  c 

4fia  crp  t-;s  share  :n 


0.4645  -C.  3707 
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whom  the  workload  was  a  concern  to  the  judges.  Professors 
B,  H,  K,  1,  and  P  all  have  high  median  factor  scores  with 
respect  to  this  common  factor.  Factor  2  might  represenr  a 
composite  effect  between  the  scales  'class  size'  and  'course 
relied  upon  prerequisites'.  Professors  B,  I  and  N  had  high 
factor  scores  with  respect  to  this  indicator.  Factor  3 
encompasses  a  general  student-professor  interaction  effect. 
Most  of  the  professors  had  the  same  spread  of  factor  scores 
except  for  professor  K.  Here  the  judges  exhibited  a  higher 
level  of  controversy  for  this  characteristic. 

E.  KESU1TS  OF  C10STEB  ANALYSIS 

A  final  means  of  looking  at  the  groupings  of  professors 
focused  on  a  cluster  analysis.  In  this  technique, 
professors  who  were  perceived  to  be  similar  to  one  another 
were  grouped  together  in  the  same  cluster.  Other  professors 
were  likewise  grouped  in  other  disjoint  clusters  so  that  no 
professor  belonged  to  more  than  one  cluster.  The  professors 
were  assigned  to  clusters  based  upon  their  interpoint 
distance  from  the  cluster  means.  An  initial  cluster  seed  is 
selected  and  the  iterative  process  continues  until  the 
observations  become  stable,  i.e.,  each  observation  settles 
down  into  a  steady  state  cluster.  It  remains  for  the 
researcher  to  decide  upon  the  number  of  clusters  that  best 
describe  the  groupings  of  professors.  One  strategy  that  was 
employed  in  determining  the  appropriate  number  of  clusters 
resulted  from  interviews  with  the  students.  Tables  13  and 
14  display  the  clusters  to  which  each  professor  belonged 
from  the  four  and  five  dimensional  solutions. 


TABLE  12 

Tafcle  of  Pactor  Analysis  Besults  on  SO F  Data 


SQP|=C3u«Se  ORGANIZAHCN 
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SGP<*  =  UIFP  ICucr  COnC£°  t  S  «AOe  unCERST  AN0A8LE 

SOP5  =  CQNf  iOEMCE  IN  INSTRUCTORS  KNOWLEDGE  IN  SUBJECT 
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SuF 7  = l NSTPUC fc«  o«EP48FD  FOR  CLASS 
SQF 5  =  l  NS  rw(jCTG°S  aaJECTlVES  *AO£  CLEAR 

SQF93 INSfflUCrQP  MAOE  COURSE  WORTHWHILE  LEARNING  EXPERIENCE 
SQFlOslNSrauCTQR  STIMUL ATEO  INTEREST  IN  Sue^ecr  AflgA 

scf i i a r ns rRuc tom  cabfo  aacur  student  progress  ano  did  his  shape  in 
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CORRELATION  matrix 

SOF  i 

SOF  2 

SOF  3 

SCFa  SOF  5 

SOF 6  SCF7 

SGF8 

SOF 9  SOF 10 

SQF  l 

SOF  l 

1 .00 

0.70 

0.60 

0.66  0.42 

0.49  0.66 

0  .69 

0.69  0.57 

0  •  o  5 

SCF2 

0.  70 

t  .00 

0.62 

0.67  0.43 

0.4J  0.60 

0.59 

0.70  0.59 

0 .60 

SQF  3 

0.60 

0.62 

t  .  00 

0.72  0.42 

0.54  0.58 

0.52 

0.66  0.57 

0.72 

SCF* 
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0.67 

0.72 

1.00  0.43 
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0. 55 
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0  .  70 

SQF  5 

0  .42 

0.  a  3 

0.42 

0.43  l .00 

0.37  0 .46 

0.  35 

0.50  0. 44 

0.46 

SOF  6 

0.49 

0.  Aj 

0.S4 

0.55  0.J7 

l  .0  a  0  .5  7 

0.44 

0.53  0.46 

0.61 

SOF  7 

0  .68 

0.60 

0.58 

0.62  0.48 

0.5  7  1  .00 

0.57 

0.63  0.S5 

0.66 

SOFd 

0.69 

0.59 

0.52 

0.55  0.35 

0.44  0.57 

1  .00 

0.60  0.50 

0 . 58 

SQF  9 

0.69 

0.70 

0  .6© 

0.70  0.50 

0.53  0 • 6  J 

0.60 

I . 00  0. 74 

0.  70 

SOF  10 

0.57 

0.  59 

0.5  7 

0 • 5d  0.44 

0.46  0.50 

0. 50 

0.74  t . 00 

0.59 

SOF  l  I 

0.65 

0.60 

0.72 

0  •  7vj  0  .46 

0.61  0.66 

0.59 

0.7Q  0.59 

l  .00 
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TABLE  11 

Table  of  Factor  Analysis  Results  on  Survey  Data 


SCALE  l-  CLASS  SIZE 

scalc  2-  amueo  «s.  rnecwsMCAL  couose 

SCALE  3=  oSAOING  POLICE 
SCALE  A=  PACE  OF  COUPSE 

SCALE  Sa  cFFOPT  PeOUIPEC  OUTSIDE  CLASS 
SC»LC  SA  CCuOSE  PEL l E  0  UPON  p hcM E CU 1 S i TE S 


caapEL*rtaN  aatsi* 


SCALE  l 

SCALE  2 

SCALt  3 

SCALE  * 

SCALE  5 

SCALc  o 

SCALE 

1 

i  .0000 

-0 . 1 32 7 

0 . 1800 

0.06*7 

0 .0*69 

-0.2815 

SCALE 

2 

-0 . 1 32  7 

t  .  ocau 

0 . 3 l  10 

0.267  l 

0.1931 

0. 2763 

SCALE 

3 

0 •  t  800 

0.3110 

1  .0000 

0 . 5  12* 

0.610* 

0. 3357 

SCALE 

4 

0 .06*  7 

0.26^1 

0.51.?* 

l  .OCOO 

0.57^1 

0. *928 

SCALE 

5 

0 . 0*69 

0.1931 

0.610* 

0 .5  75  1 

l  .0000 

0 . 50*0 

scale 

A 

-0 . 28  t5 

0.2  762 

0.3357 

0.492c 

0 .50*0 

1  .  0000 

factob  patteon 


NO  MOTATtON 

□  9  rt-OGONAL 

WOT  A  r  I  ON 

FAcrcfi  1* 

FACTOR  2 

FAC  row  l 

F4CTQW  2 

SCALE  l 

-0 .02185 

0.71926 

0  .2*550 

0. 366  14 

SCALE  2 

0 . *o5 l 9 

-0 . 305*7 

0  .3  7584 

-0 . *  3298 

SCALE  3 

0 . 7748  3 

0 . 32461 

0  .8  2506 

0. 0o6  12 

SCALE  * 

0.80*53 

0  .  1  l  096 

0 .80215 

-0. 12697 

SC  *lE  * 

0.82597 

0.151*5 

0  •  3  J4*  2 

-0.  094*  3 

SC-U-E  6 

0 . 720  16 

-0 .* l 322 

0  .569*9 

-0 • 60*20 

eigenvalues 

OF  THE  COROtLATlON  M4T91X: 

TQT  AL  -8 . 0U000 

AVERAGE 

s l .00000 

t 

2  3 

4 

5 

6 

E I GcNV aluE 

2 . o8*4 

1.2*97  0.8361 

0 • *9  3  3 

0  .*032 

0 .3283 

0£FFe SENCE 

l  •  *  347 

0 • *  1 36  0 . 3  3  75 

C.J9  i,2 

J  .0  7«.  9 

PQCF09  7  l  ON 

0 . *4 74 

0 .2033  0.1393 

C.0831 

0 .0672 

0 .05*  7 

Cunul A  r I Vfe 

0  .  *4  74 

0.0557  0.7950 

0.8731 

0 . 9*5  3 

l .0000 
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one  on  each  set  of 


separate  factor  analyses  were  conducted, 
bipolar  scales.  Each  will  be  discussed  separately. 

The  factor  analysis  conducted  on  the  six  bipolar  scales 
used  in  the  computer  survey  yielded  the  results  shown  in 
Table  11  .  In  addition  to  the  correlation  matrix,  the  unro¬ 
tated  factor  loadings  and  the  orthogonally  rotated  factor 
loadings  appear  in  the  table.  variables  with  factor  load¬ 
ings  that  are  close  numerically  suggest  a  common  interaction 
or  measure.  It  seems  as  though  two  common  factors  are 
present  in  the  six  variables  based  upon  the  factor  loadings. 
The  first  factor  explains  44  percent  of  the  variation  in  the 
data.  This  factor  seems  to  combine  the  affect  of  ’grading 
policy’,  ’effort  reguired  outside  class’,  and  ’pace  of 
course’.  The  second  factor  appears  to  be  a  composite  effect 
of  ’class  size’  and  'course  relied  upon  prerequisites'.  The 
scale  'class  size'  has  a  high  positive  factor  loading 
whereas  the  scale  'course  relied  upon  prerequisites'  has  a 
fairly  high  negative  loading.  This  would  seem  to  make  sense 
since  those  classes  taken  early  in  the  curriculum  tended  to 
be  large  and  the  earlier  courses  did  not  usually  require  a 
significant  amount  of  prerequisite  courses. 

The  results  of  the  factor  analysis  on  the  SOF  data 
yielded  the  correlation  matrix  and  factor  loadings  in  Table 
12  .  From  this  factor  analysis,  we  see  that  only  one  common 
factor  accounts  for  the  variance  in  the  data.  This  is 
reasonable  since  most  of  the  bipolar  scales  from  the  SOF 
forms  are  highly  correlated.  This  factor  could  describe  the 
student-professor  interaction  effect  discussed  earlier. 

figures  4.3,  4.4,  4.5  illustrate  boxplots  of  factor 
scores  for  each  professor  for  all  three  factors.  Factor  1 
again  is  an  indicator  of  the  combined  effect  of  'grading 
policy',  'pace  of  course',  and  ’effort  reguired  outside 
class'.  It  might  be  condensed  into  a  general  workload  index 
with  a  high  factor  score  indicating  those  professors  for 
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TABLE  10 

Table  of  7alaes  for  E2  Dae  to 

Sigh  Multicollinearity 

a  check  pQp  mut  i coll  f  ne ap  i  rr  among  independent  vauiabi.es  resulted  in  the 
FOLLOWING  VALUES  OF  R-SGUAREQ. 

computer  bipolar  scales 

0€°ENO€  n  r  VAPIA0U 

RESULTING  R-SOUAREO 

CLASS  SIZE 

0.526 

fNEOPETtCAL  V£.  APPLIED 

0.2*5 

GUAOING  POLICY 

0.95  1 

PACE  OF  COURSE 

0.978 

effort  required  aursioe  class 

0.826 

course  pelieo  upon  prereou  is i tes 

0  .39  t 

SOF  BIPOLAR  SCALES 

OE°ENOENT  variable 

RESULTING  R-SCUARED 

COURSE  ORGANIZATION 

0.961 

TIME  IN  CLASS  SPENT  EFF£C M VELY 

0.9a  I 

INSTRUCTOR  KNQVS  vh£N  STUDENTS  CON • T 

unoerstano  material 

0.905 

DIFFICULT  CONCEPTS  MA OE  UNOERST ANOAeLc 

0.9A7 

CONFIDENCE  in  INSTRUCTORS  KNOWLEDGE  IN 
SUBJECT 

0 . 762 

f'ELT  pnee  ra  ASK  QUEST  t  GNS 

0.9A  l 

INSTRUCTOR  PREPARED  FOR  CLASS 

a. <563 

INSTRUCTORS  OBJECTIVES  MACE  CLEAR 

O.BA0 

INSTRUCTOR  MAOe  COURSE  aOSThkhILE 

0.986 

learning  EXPERIENCE 

INSTRUCTOR  stimulated  interest  in 
SUBJECT  AREA 

0.930 

INSTRUCTOR  CAREO  ABOUT  STUOENT 

PROGRESS  anO  DID  HlS  SHARE  IN 

HELPING  TO  LEARN 

0.960 

* 

TABLE  9 

Output  from  Stepwise  Regression  Procedure 


atsu-fs  f an«  stepfISe  secoessiOF  qf  bipolab  scales  from  intepactive 


SURVEY  ON  INSTRUCTOR  OVERALL  PERFORMANCE 

ORDER  IN  wHtCH  VARIABLE  REGRESSION 

ENTERED  THE  MODEL  COEFFICIENT 

s  taaoaro 

ERROR 

T-v  aluE 

%  OF 

VARl  AT  ION 
SXPL  A l NED 

theoretical  vs* 

applied 

0.214 

0*26  7 

0  •  5d  A 

0*044 

grading  policy 

-0.5IT 

l.AOl 

*0*368 

0.016 

EFFORT  REQUIRED 
CLASS 

outsioe 

0  *AB5 

0.647 

-0.750 

0.024 

RAC?  OF  COURSE 

2.392 

3*225 

0.  7*2 

0.012 

COURSE  RELIED  UPON  RRE- 
REQUISI TES 

—0  *46  3 

0.644 

-0.573 

0*041 

CLASS  SIZE 

0  *028 

0*695 

0.04  2 

0.00  1 

x  qf  instructor  overall  performance  explained  by  scales  0.13a 


RESULTS  FRO*  STEPWISE  REGRESSION  OF  BIPOLAR  SCALES  FROM  SOF  FORMS 

QN  INSTRUCTOR  OVERALL  PERFORMANCE 


QROER  IN  WHICH  VARIABLE  REGRESSION 

ENTERED  the  FQOEL  coefficient 

STANDARD 

error 

t-value 

X  OF 

VAR  I  AT  ION 
EAPL At NEO 

COURSE  ORGANIZATION 

!  *070 

0.4  11 

2. 60S 

0.737 

INSTRUCTOR  KNOWS  WHEN 

students  UNDERSTAND 

*4  TERIAL. 

I  .234 

0.224 

5.498 

0.067 

FELT  FREE  TO  ASK  QUES¬ 
TIONS 

-1 *819 

0.339 

-5. 363 

0.036 

CONFIDENCE  IN  INSTRUCTORS 
KNOWLEDGE  in  SU8JECT 

2.062 

Q  •  «40 

4.69  1 

0.03  1 

INSTRUCTOR  STIMULATED  IN¬ 
TEREST  IN  SUflJECT  AREA 

-0.415 

0. 416 

-0.996 

0.015 

TIME  IN  CLASS  SPENT  EFFEC- 
T| VELY 

-1  .000 

0.250 

-2.859 

0.031 

instructor  careo  about  stu- 
OENT  PROGRESS  anO  OlD  h|S 
SHARE  IN  HELPING  TO  LEARN 

1  .539 

0.629 

2.444 

0.020 

INSTRUCTOR  made  COURSE 
WORTHWHILE  learning 

EXPER IENCE 

-1  .795 

0.e79 

-2 .04  I 

0.033 

INSTRUCTOR  PREPARED  FOR 
CLASS 

0  .679 

0.4QQ 

l  .696 

0.004 

OIFFICULT  CONCEPTS  “AOE 

unoerst ANO able 

0  .575 

0.240 

1  .691 

0.01  1 

INSTRUCTORS  OBJECTIVES  mage 

0.107 

0.227 

0.470 

0.00  t 

CLEAR 

*««««« ««*««« «»«« *««<»**  *$***$*«  a#s##«mm#«»was8w*#«a 
X  QF  INSTRUCTOR  OVERALL  PERFORMANCE  EXPLAINED  ST  SOP  SCALES  0.986 


multicollinearity.  Multicollinearity  simply  means  that  one 
or  more  so  called  independent  variables  are  highly  corre¬ 
lated  with  another  independent  variable  or  is  a  linear 
combination  of  a  number  of  the  independent  variables.  The 
problem  with  high  multicollinearity  is  that  the  estimates 
for  the  regression  coefficients  become  unreliable  from  one 
sample  to  the  next.  Our  confidence  in  our  ability  to  deter¬ 
mine  the  effect  of  an  independent  variable  withers.  To  show 
that  high  multicollinearity  exists,  we  need  to  regress  each 
independent  variable  over  all  the  other  independent  vari¬ 
ables  to  see  if  any  are  a  linear  combination  of  the  others. 
Table  10  shows  the  values  of  K2  obtained  by  regressing  each 
independent  variable  over  the  others  for  bo£h  models. 

Several  options  are  available  when  confronted  with  high 
multicollinearity  as  we  were  in  this  study.  One  solution  is 
to  increase  the  sample  size.  This  turned  out  not  to  be  a 
useful  alternative  since  our  sample  size  was  fixed.  Another 
strategy  is  to  combine  several  variables  that  are  highly 
correlated  into  a  single  indicator  as  long  as  it  makes 
sense.  This  is  possible  for  several  scales  on  the  SOF  form 
which  are  highly  correlated.  A  third  alternative  is  to 
discard  those  variables  which  are  linear  combinations  of  the 
others  and  are  the  cause  for  the  high  multicollinearity. 
After  discarding  the  offending  variables,  a  new  regression 
eguation  can  be  created  and  a  check  for  statistical  signifi¬ 
cance  made  anew. 

D.  BESOITS  OF  FACTOB  AHA LI SIS 

Factor  analysis  supposes  that  some  common  factors 
smaller  in  number  than  the  originally  observed  variables, 
account  for  the  covariation  of  the  originally  observed  vari¬ 
ables.  Factor  analysis  assumes  a  linear  causal  relationship 
similar  to  linear  regression  analysis.  For  this  study,  two 
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TABLE  8 

fledian  Values  of  Overall  Teaching  Perforoance 

Sixteen  Professors 
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Figure  4.2  Changes  in  the  Value  of  the  Multiple  Correlation 
Coefficient  as  a  Function  of  Oinension. 
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DIMENSION  SIZE 


Stepvise  regression  was  utilzed  in  order  to  determine 
which  scales  or  independent  variables  explained  the  greatest 
proportion  of  variance  in  teacher  performance.  The  API 
function  STEEG,  was  used  to  compute  the  order  in  which  the 
independent  variables  were  to  enter  into  the  regression 
model.  The  output  from  STEEG  is  displayed  in  Table  9  for 
both  the  six  scales  asked  in  the  computer  survey  as  well  as 
the  scales  from  the  SOF  forms.  The  output  indicates  the 
order  in  which  the  variables  entered  the  model,  the  coeffi¬ 
cients  associated  with  each  independent  variable,  the  stan¬ 
dard  error  associated  with  each  coefficient,  the 
corresponding  t- value  and  the  proportion  of  the  variation  of 
the  dependent  variable  accounted  for  by  the  independent 
variable.  We  see  that  the  coefficient  of  multiple  determi¬ 
nation,  R2,  is  low  for  the  six  bipolar  scales  used  in  the 
computer  survey.  In  all,  these  scales  account  for  only  13.8 
percent  of  the  variation  in  the  dependent  variable,  overall 
instructor  performance.  However,  the  stepwise  regression 
performed  on  the  eleven  bipolar  scales  from  the  SOF  forms 
indicate  a  coefficient  of  multiple  determination  of  .986. 
This  set  of  scales  explains  more  of  the  variation  in  the 
judges  responses  to  overall  instructor  performance  than  does 
the  previous  set  of  scales.  In  particular,  'course  organi¬ 
zation'  appears  to  have  a  significant  impact  on  students 
perception  of  teacher  performance  since  it  accounts  for  over 
73  percent  of  the  variation  alone. 

One  problem  that  concerned  us  dealt  with  the  fact  that 
for  the  SOF  data,  even  with  a  value  of  R2  egual  to  .986, 
most  of  the  independent  variables  were  statistically  insig¬ 
nificant.  Even  the  scale  'course  organization',  which 
accounted  :or  73  percent  of  the  variation  in  overall 
performance  ,  proved  to  be  statistically  insignificant  at 
the  0.05  level.  A  high  value  of  R2  and  statistically 
insignificant  variables  usually  is  symptomatic  of  high 
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TABLE  7 


Regression  Weights  and  Hultiple 
Coefficients  for  tne  Bipolar  Scales 


Correlation 
in  5  Dinensions 
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felt  free  to  ask  questions;  4} instructor  made  course  a 
worthwhile  learning  experience  and  5)  instructor  cared  about 
student  progress  and  did  his  share  in  helping  to  learn. 
Dimension  2  seems  to  describe  information  concerning  how 
instructors  are  perceived  as  being  organized  or  prepared  for 
their  course.  The  scales  with  high  multiple  correlation 
coefficients  that  load  heavily  upon  this  dimension  are  1) 
course  organization;  2) time  in  class  spent  effectively  and 
3)  instructors  objectives  are  made  clear. 

In  addition  to  the  bipolar  scales  used  for  the  interac¬ 
tive  survey  and  the  SOF  forms,  each  judge  was  asked  to 
provide  a  rating  of  overall  teaching  performance  for  each 
professor.  The  median  values  for  each  professor  were 
obtained  and  are  tabulated  in  Table  8  .  Although  we  have 
kept  the  analysis  of  the  interactive  survey  separate  from 
the  SOF  data,  we  were  interested  in  the  correlation  between 
professor's  overall  performance  judged  on  two  separate 
occaisions.  The  first  occaision  occurred  when  the  judge 
completed  the  SOF  form  for  the  course  taught  by  the 
professor.  The  second  occaision  was  in  conjunction  with  the 
interactive  survey.  The  correlation  between  ratings  of 
overall  performance  for  the  professors  was  .58,  lower  than 
expected.  Suggestions  as  to  why  this  correlation  is  low 
include  the  effect  of  time.  Judges  may  be  less  able  to 
evaluate  a  particular  instructors  overall  performance  as 
time  goes  by.  Also,  students  may  change  their  opinions  of 
teachers  overall  performance  after  having  seen  a  professor 
teach  a  variety  of  different  classes.  For  example,  a 
student  who  enjoys  applied  courses  might  rate  a  professor 
differently  after  having  had  him  f or  a  Stochastics  Models 
course  than  he  would  for  say  an  applied  course  like  Test  and 
Evaluation.  In  any  event,  we  did  expect  the  correlation  on 
overall  performance  to  be  higher  than  it  turned  out  to  be. 


42 


1/ 


TABLE  6 

Regression  Heights  and  Multiple  Correlation 
Coefficients  for  the  Bipolar  Scales  in  4  Dimensions 


POSITIVE  POLES 
sating  SCALES 


(JF 


COMPUTER  81  POLAR  SCALE: 
l  •  CLASS  SIZE 


MGRKALIZFn  REGRESSION  COEFFICIENTS 
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fact  that  a  course  was  perceived  as  being  theoretical  or 
applied  most  closely  describes  the  effect  of  the  professors 
ultimate  positioning  with  respect  to  dimension  2.  Likewise, 
the  scale  ’course  relied  upon  prerequisites’  loads  heavily 
on  dimension  4  with  a  regression  weight  of  -0.7270.  This 
appears  to  suggest  that  dimension  4  is  most  nearly  explained 
by  how  the  judges  perceived  how  each  course  relied  upon 
other  prerequisite  courses.  The  regression  weights  have  a 
geometrical  interpretation  as  the  cosine  of  the  angle 
between  the  dimension  upon  which  it  loads,  and  the  associ¬ 
ated  scale.  Unfortunately,  the  two  scales  with  the  next 
highest  multiple  correlation  coefficients,  ’pace  of  course* 
and  ’grading  policy’,  load  heavily  upon  dimensions  2  and  4 
as  well.  They  probably  contribute  to  the  explanation  of 
those  dimensions  also.  Not  one  of  the  scales  with  a  high 
multiple  correlation  coefficient  loads  heavily  upon  dimen¬ 
sions  1  or  3.  These  dimensions  are  left  unexplained  and  new 
scales  are  needed  to  determine  an  appropriate  explanation. 

A  similar  analysis  was  conducted  for  the  bipolar  scales 
used  in  the  SOF  forms.  Here  the  scales  with  the  highest 
multiple  correlation  coefficients  were  1)  instructors  objec¬ 
tives  made  clear;  2)  instructor  cared  about  student  progress 
and  did  his  share  in  helping  to  learn;  3)  course  organiza¬ 
tion  and  4)  time  in  class  spent  effectively.  Again,  the 
same  problem  occurred  here  as  in  the  analysis  of  the 
previous  set  of  scales.  Almost  all  those  scales  with  the 
highest  multiple  correlation  coefficients  load  heavily  on 
only  two  of  the  four  dimensions,  namely  dimension  1  and  2. 
Dimension  1  appears  to  convey  information  about  how  each 
professor  cares  or  interacts  with  his  students  on  a  personal 
level.  The  bipolar  scales  with  high  multiple  correlation 
coefficients  that  load  heavily  on  this  dimension  are,  1) 
instructor  knows  when  students  don't  understand  the 
material;  2)  difficult  concepts  were  made  understandable;  3) 
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Figure  4.5  Boxplot  of  Factor  Scores  for  Factor  3 
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TABLE  14 

Table  of  Clusters  for  Five  Dimensional  Solution 
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V.  SUMMARY  OF  RESULTS 


A.  REVIEW  OF  THE  ESSIHTIAI  ITEMS 

The  purpose  of  this  study  was  to  uncover  information 
about  students  perceptions  of  teacher  performance  from  data 
gathered  via  a  computer  interactive  survey  and  SOF  forms. 
The  control  group  for  the  interactive  survey  was  the 
Operations  Research  section  graduating  in  March  of  1985. 
The  data  consisted  of  proximity  information,  how  similar  or 
dissimilar  professors  were  perceived  as  being,  and  rankings 
on  bipolar  scales.  Bipolar  scales  were  chosen  from  SOF 
forms  and  suggestions  from  previous  Operations  Research 
students.  Separate  analyses  were  conducted  on  both  sets  of 
scales.  The  level  of  the  data  was  assumed  to  be  interval 
scale  in  order  to  utilize  the  statistical  methods,  involved 
in  the  analysis. 

Multidimensional  scaling  was  used  as  the  primary  means 
of  evaluating  characteristic  differences  among  professors. 
A  monotone  relationship  among  similarity  data  was  the 
primary  constraint  used  in  determining  a  final  spatial 
mapping  of  professors  coordinate  positions  in  multidimen¬ 
sional  space.  The  goodness-of-fit  criterion  used  to  measure 
the  degree  to  which  the  data  conformed  to  the  monotonicity 
requirement  is  known  as  stress.  In  essence,  a  value  of 
stress  between  .05  and  .1  would  indicate  a  very  good  fit. 
Unfortunately,  the  stress  value  associated  with  a  four 
dimensional  interpretation  of  our  data  was  .  242,  indicating 
a  less  than  good  fit.  Other  methods  helped  guide  the  choice 
of  dimensionality. 

Aside  from  visual  inspection  of  the  spatial  mappings, 
multiple  linear  regression  analysis  was  used  to  determine 
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the  most  important  characteristics  that  appeared  to  set  the 
professors  apart  from  one  another.  This  was  done  by 
regressing  the  median  values  of  the  bipolar  scales  over  the 
coordinate  positions  of  each  professor  in  four  space, 
obtained  by  the  MDS  program,  KYST.  The  choice  of  the  most 
important  characteristics  follows  from  the  bipolar  scales 
with  high  multiple  correlation  coefficients.  The  multiple 
correlation  coefficients  associated  with  all  bipolar  scales 
were  lower  than  what  was  hoped  for.  The  correlation  coeffi¬ 
cients  were  typically  around  .5.  For  those  characteristics 
deemed  important,  high  regression  weights  determined  exactly 
which  dimension  the  associated  scale  most  nearly  repre¬ 
sented.  For  the  bipolar  scales  used  in  the  interactive 
survey,  the  scales  'applied  vs  theoretical  course'  and 
•course  relied  upon  prerequisites '  proved  to  have  the 
highest  correlation  coefficients.  These  scales  most  nearly 
explained  dimensions  2  and  4.  The  other  two  dimensions  were 
more  difficult  to  explain  since  no  scale  with  a  high 
multiple  correlation  coefficient  loaded  heavily  on  them. 

For  the  set  of  scales  obtained  from  the  SOF  forms,  a 
similar  result  occurred  with  the  scales  having  high  multiple 
correlation  coefficients  loading  heavily  on  only  two  dimen¬ 
sions.  The  scales  with  the  highest  correlation  coefficients 
here  were  1)  instructors  objectives  made  clear,  and  2) 
instructor  cared  about  student  progress  and  did  his  share  in 
helping  to  learn.  There  seemed  to  be  two  indicators  coming 
from  the  SOF  forms.  One  indicator  seemed  to  focus  on 
instructor  organization  and  preparation.  The  other  indi¬ 
cator  involved  a  student-instructor  interaction  effect. 
Basically,  how  did  the  judges  perceive  the  instructor  as 
caring  about  the  students  progress  in  the  course?  This 
effect  seemed  to  be  corroborated  in  the  factor  analysis. 

In  addition  to  the  bipolar  scales,  students  were  asked 
to  rate  the  professors  on  an  overall  performance  scale. 
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Multiple  and  stepwise  regression  efforts  were  done  using  the 
overall  performance  evaluations  obtained  from  the  SOF  forms 
and  at  a  later  date  coincident  with  taking  the  computer 
survey.  The  important  information  gleaned  from  the  regres¬ 
sion  analysis  suggested  that  the  six  bipolar  scales  used  in 
the  computer  survey  did  a  poor  job  in  explaining  students* 
perceptions  of  instuctor  overall  performance.  The  coeffi¬ 
cient  of  multiple  determination  for  the  regression  model  was 
a  mere  .138.  Stepwisa  regression  analysis  indicated  that 
the  scale  ’applied  vs.  theoretical  course*  accounted  for 
most  of  the  variation  in  the  dependent  variable,  overall 
performance,  for  its  set  of  scales.  Thus,  it  would  seem 
that  whether  students  perceived  a  course  taught  by  a 
professor  as  being  applied  or  theoretical  had  a  more  signif¬ 
icant  bearing  on  the  overall  performance  of  the  professor 
than  the  other  scales  in  that  set.  In  any  case,  none  of  the 
scales  proved  to  be  statistically  significant  at  the  .05 
level. 

The  stepwise  regression  performed  on  the  SOF  data 
yielded  a  coefficient  of  multiple  determination,  F.2  ,  of 
.986.  This  rather  high  value  of  E2  seemed  to  suggest  that 
the  scales  used  in  the  SOT  forms  more  nearly  explain  the 
variation  in  overall  performance  than  do  the  scales  used  in 
the  computer  survey.  ’Course  organization'  accounted  for  74 
percent  of  the  explained  variation.  Even  with  this  high 
value  of  R2  ,  most  of  the  coefficients  of  the  independent 
variables  turned  out  to  be  statistically  insignificant  at 
the  .05  level.  This  indicated  that  high  multicollinearity 
existed  among  the  scales.  A  check  for  multicollinearity 
proved  positive  in  both  sets  of  scales.  Each  independent 
variable  was  regressed  over  the  other  independent  variables 
and  high  values  of  B2  resulted.  The  multicollinearity 
problem  suggested  that  a  number  of  the  scales,  particularly 
on  the  SOF  forms,  be  combined  into  one  scale  or  measure. 
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In  addition  to  regression  analysis,  a  factor  analysis 
was  done  on  both  sets  of  scales  and  three  separate  factors 
were  obtained.  Factor  1  was  composed  of  the  three  scales, 
♦grading  policy’,  ’effort  required  outside  class’  and  ’pace 
of  course*.  Apparently,  the  judges  found  these  scales  to 
interact  consistently.  Factor  2  appeared  to  describe  a 
composite  effect  between  'class  size'  and  ’course  relied 
upon  prerequisites’.  The  factor  loading  was  positive  on 
’class  size'  and  negative  on  ’course  relied  upon  prerequi¬ 
sites'.  One  might  infer  that  the  larger  the  class  size,  the 
less  that  course  was  perceived  as  requiring  prerequisite 
courses  in  Operations  Research.  This  appeared  to  be  true 
since  most  of  the  larger  classes  occurred  in  the  beginning 
of  the  curriculum.  Factor  3  seemed  to  describe  a  student- 
professor  interaction  effect.  The  correlations  among  the 
variables  in  this  set  were  high  contributing  to  high  factor 
loadings  on  nearly  all  variables. 

A  disjoint  cluster  analysis  was  conducted  on  the  coordi¬ 
nates  generated  from  the  multidimensional  scaling  algorithm. 
Each  professor  was  grouped  into  one  and  only  one  cluster. 
Exit  interviews  with  students  suggested  that  five  clusters 
appeared  to  be  an  appropriate  number  of  groups. 

B.  COHCIOSIONS  AND  BECCOHENDATIONS 

The  multidimensional  scaling  technique  seemed  to  provide 
a  weak  explanation  of  instructor  groupings.  The  reasons  for 
this  may  be  several.  First,  a  linear  causal  relationship 
nay  not  be  appropriate  in  describing  students'  perceptions 
of  instructor  performance.  Certainly  the  six  bipolar  scales 
used  in  the  interactive  survey  were  not  powerful  explanatory 
variables.  However,  there  seem  to  be  one  or  two  strong 
indicators  among  the  scales  used  in  the  EOF  forms. 
Specifically,  a  student-instructor  interaction  effect  and  an 
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instructor  organizational  and  preparation  effect  appear 
dominant.  flhat  needs  to  be  done  in  the  future  is  to  obtain 
further  scales  or  characteristics  from  Operations  Besearch 
students  possibly  during  exit  interviews.  Also,  one  might 
suggest  that  another  look  be  given  to  changing  the  current 
SOF  fora  as  it  stands  by  combining  some  of  the  highly  corre¬ 
lated  variables  and  adding  ^  characteristics  that  later 
prove  meaningful.  However,  it  should  be  noted  that  what 
aight  be  considered  an  important  characteristic  in 
describing  instructor  performance  for  one  student  group  may 
prove  to  be  less  important  for  another  group. 
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*  STACK  SELECTEO  PROFESSORS. 

EUOEXSC2  SURVMAS 

*  PLACE  SELECTED  PROFESSORS  IN  TEMPORARY  CMS  FILE 

• 

XEDIT  OR  DATA  A  (NGFPOF 

*  STAPT  interactive  QUESTIONING. 

*  OUTPUT  PGSuLTS  TO  FILE  F~02F0C1 

FILEOEF  01  0 1 S  K  OR  DATA  A 


« 

* 

• 

SUR  V3 

ERASE  OR  CATA 

* 


A  MOCULE  (5.URV3  MODULE )  WAS  CREATED  TO  ALLEVIATE  THE 
PROBLEM  OF  EACH  STUCEiNT  NOT  HAVING  THE  NECESSARY 
FORTRAN  COMPILER.  THr  PROGRAM  (SURV3  F0°  TRAN )  IS 
LOADED  AND  BEGUM  MORE  EFFICIENTLY  WITH  THIS  DEVICE. 


FTNC  CUT  RESPOMOcNT  •$  USERID. 
WITH  THE  IP?  FUNCTICN  USSRID. 


USERID 

* 

* 

* 

CRP  40  VARS  CDUMMY  CUSEP.ID  COUMHY  CD  ATc  CTI  ME  SDAY 


PLACE  USERID,  DATE,  5  TIME  IN 
EXEC 2  VARIABLES. 


GIVE  OUTP'r  RESULTS  FILENAME  OF 
THE  RESPONDENT  AMD  SEND  FILE  TO 
3 1 77P  WITH  01  SK  DUMP. 

3177P  IS  THE  PROGRAMMERS*  CURRENT  ACCOUNT  *. 


* 

« 

* 

* 

* 

R91AME  FILE  FTQ2F00 1  A  SUSERIO  PROFS  A 

*  EACH  FILE  IS  GIVEN  FIXED  LOGICAL 
RECORO  LENGTH  OF  30  COLU.MNS. 

CCPY'ILE  OJSERID  PROFS  A  I RECFM  F  LRECL  30 

*  «*  RFMcMB'R  TO  SPOOL  PUNCH  TO  RESEARCHERS  USERID  * 
CP  SPOOL  PUNCH  TO  3177P 

*  FILES  ARE  SPOOLED  AND  PUNCHED  TO  RESEARCHER 
0 IS  K  DUMP  SUSERIO  PPCFS 

OP  SPOOL  PUNCH  CFF 

*  WHEN  SURVEY  IS  COMPLETED.  THF  FILS  OF  RESPONSES 

*  IS  ERASED  FPOM  THF  RESPONDENTS  A  DISK 
ERASE  CUSER  10  PROFS 


DIMENSION  2 


c 
c 
c 
c 
c 
c 
c 
c 

Ug=  panel  survi 

£C2 

cos 

£C4 
£  C5  ' 

CC6 
£07 
CCS 
£09 
COLO 
CC1X 
£012 
£013 

£014 
£015 
£016 
£C1  7 
£018 
£015 
£C20 
£021 
£022 
£C2  3 
£02  4 
£02  5 
£02  6 
£02  7 
£02  8 
£C2  9 
£03  C 
£03  l 
£03  2 
£033 
£034 
£035 
£036 
£03  1 
£038 
CC3S 
DISPLAY 

CSTACK  I  01601 


SUPV',AS"  AC^SVlKsPr^S?UP(Op|SSCP|  1  S UP. V l  PAN=U 


£  STACK 
65. ACK 
6  ST  ACK 
6STACX 
cstack 
cstack 
cstack 

£ST  ACK 

ACK 

CSTACK 

CSTACK 

CSTACK 

CSTACK 

CSTACK 

CSTACK 

CSTACK 

CSTACK 

CSTACK 

CSTACK 

CSTACK 

CSTACK 


02C5! 

:  03  £0  3 
04  £04 
0560  5 
06  606 
0760  r 
I  08603 
I  09  60  5 
I  LOCO  1  G 
I  UCOU 
I  126012 
I  1363 1  3 
f  146014 
1  156015 
I  166016 
I  1760 1  7 
{  186018 
t  L96015 
I  20  £0  20 
I  216021 
I  22 £022 
I  23 £023 
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£3.  ACK 
fiSTACK 

I 

24  £0  2  4 

I 

25£02  5 

1ST  4CK 

I 

26 £024 

£  STACK 

I 

27  i  0  2  7 

£  ST  ACK 

I 

2BfiOI£ 

£STACK 

I 

29  ao  2  9 

fiSTACK 

I 

30  0030 

£3T  ACK 

I 

31  £0  3  L 

fiSTACK 

I 

32  £0  3  2 

fiSTACK 

I 

33fi03  3 

fiSTACK 

I 

34£D  3  4 

fiSTACK 

I 

35 £025 

fiSTACK 

I 

36fi036 

fiSTACK 

T 

37fiD3  7 

fiSTACK 

i 

38  £0  3  8 

fiSTACK 

i 

39&C39 

fiSTACK 

FIL" 

txsiia  issiixxs&siaaxtxxin  aa  as  9u»  as  aas  aa  3  a  aa  a  a  a  aaaa  XU  a  a  a  a 

c 

C  SURV1  PA MEL 

C 

C  THE  ^CLLDwIAG  PAN.'L  CQfIT  41  MSA  LIST  OF  39  PROFESSORS 

C 

-,THS  FOLLOWING  3URV-Y  IS  OSSIGNfC  TC  COLL'CT  DAtA  on  students* 
-.PFRCEPT  IS  IS  "'F  INSTRUCTS  EFFECTIVENESS.  E'ITER  THE  VALUE  l 
-.IN  THE  UNOERSCORSC  PCSI^IOf!  TO  Tt-c  LEFT  OF  EACH  PRQFSSSOR  FROM 
-WHOM  YOU  MAV-_  T AKi.tl  AT  LEAST  QN5  COJP.SF. 


_a-ALAN  HASHeuPN 
_i-G=RALD  3PQWN 
_S-FAIJL  FIGCHiECK 
_a-.0DNAL3  GAVrR 
_i-,G  ILBE  F  T  HCWaRO 

_j-?at°ich  jAcaas 

_5-«iL3NN  L  [  NCSA  Y 
_3-.00UGLAS  NEIL 
_i-GARY  JOSCK 
a->SLSS  RISHARQS 
_9-.REX  SHU1CE 
_«-» J  C5SPH  STEWART 
_i-KSVIN  HO CO 

-CEPPESS  THF  INTER  KEY  WHEN  F1MISH5D 


_3-.ALVIN  AN  CRUS 

_3->jAMer  eagle 

“I-MFAGLE  FCRRF  '* 
_3-JAMeS  HAFTMAN 
_a—WAYNF  HUGHES 
a— HAkOLO  LARSON 
“A-*AL.aN  TCMASr.  RS 
_3-.SA.NUFL  PARRY 
_S-GARY  PORTER 
,3-PAT  SANQC2  „ 
a— AR  MAMOD  SCLTS 
_3-TIM  SULLIVAN 
_3->JAM?j  YEE 


_S-OON4LO  3ARR 
_3-JAMES  .•  S  A  R  Y 
_S-J«CK  GAFFOPO 
3-.T1+0MAS  HO  I  V I  K 
_3-CHARLSS  HUTCHINS 
_«-PETER  LEWIS 
_S-PAUL  MILCH 
_3~HARCHMAN  PERRY 
_S-«GBSRT  R=AD 
_ 3— BRUNO  S HU8SRT 
_3— HfCHASL  SOVEREIGN 
.i-JAOSS  TAYLOR 
IS-PSTER  23HNA 


aaa  aa  =  a  a  aaaaaaaaaaaaaaaa  aa  aaaaaaa  aaaa  aaaaaaaaaaaaaaaaaaaaa  aa 

SURV3  FORTRAN 


VARIABLES  CECINFC 

I,J-  THESE  INTEGER  VARIABLES  ARE  USED  AS  SIMPLE  COUNTERS  THROUGHGU 
THR  3PCGRAM 

N  -  REFERS  TO  THE  NUMBER  OF  PROFESSORS  USED  ON  THE  PANEL  ( SURV 1) 

C  -  REFERS  TO  ThF  T-Al  NUMBER  CF  PROFESSORS  IN  THE  SUBS?T  PERTAIN 
IMG  TC  EACH  INCIVIDUAL  RESPONDENT 

ci  -  equaies  to  th-  value  c-i  fcp  each  respondent,  used  to  help 

LOOP  THE  RES° INDENT  tHRU  THE  PAIRWISE  COMPARISONS 
A  -  INTEGER  COUNT'S  SIMILAR  TQ  I  AMD  J. 

6(39  1-  DEFINES  THE  R=S?0N0E'iTS  UM0UE  SUBSET  OF  PROFESSORS 
INO.ERRCOH  -  VARIABLES  USED  IN  AN  ERR  CP  HANDLING  SUR^UTINE 

ERR  Sc"  fou.no  IN  VS  FORTRAN  MANUAL.  HEL°3  T0 
MAKE  THE  SURVEY  USER  FRIENDLY  BY  CATCHPKJ  !N- 
CCPPFCT  RESPONSES,  I.E.  HITTING  THE  EN TOR  KEY 
WHEN  i  NUMERIC  WAS  REQUESTED. 

R (39  , 391-  THF  TWO  DIMFNSIONAL  ARRAY  OF  RESPONSES  TO  SIMILARITY 

3ATA  FOR  THE  RESPONDENT.  TH?  REASON  IT  (S  DIMENSIONED 
AT  39  IS  3ECAUSS  OF  THF  NUMBER  CF  PROFESSORS  IN  THE 
P  ANFL 

8PS(7, 391— SIMILAR  TO  ABOVE  EXCEPT  THAT  IT  STORES  THt  RESPONSES  T0 
THE  8IPCLAR  SCALES 
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<J<JOO<JOOO  OOO  OOO  -O'"  MOOO 


L  1ST  (  39)  -  1  CHAR  1  CTEF  ARRAY  Cc  TF’1  PROFESSORS  lA.M^S 
A ( 39 )  —  i  C^A»ACTeP  aRPAY  that  STORES  ’HP  CHARACTER  l  3?  A  BLANK 
OEP  E NO I No  UPON  HOW  THE  R5SFONOSNT  dOMPLETED  THE  PANEL  OF 
PP.CF  "SSQRS 

Y-  A  CHAP.ACfFE  V  APT  ABLE  THAT  CHECKS  T*)  SEE  IF  THE  RESPONDENT  ANSWERED 
CCRPFCTLY  TO  CONTINUE  ON  WITH  *H=  SURVEY 

TNTFG’E  T,.N,C,  J»Z,CI,0(  39  I.INC.SRRCOM 
PEAL  R(J9,39I , BPS  1 7 , 39 ) 

CHARACT ;R»20  LIST (39) 

CHARACT:R»4  A( 39) 

CHARACT :  R  *1  Y 

common/ •  rp.com  zinc 

EXTERNAL  PY6RR 

CALL  SR9SET  (215, 256 »-l«l, MY SR R,Q) 

REMEMBER  Wc  USED  39  PROFESSORS  IN  QUP  PANEL  ISURV1) 


INITIALIZE  APRAYS  P  AND  BPS  TO  O.C 

OC  5  1*1, N 
CO  o  J*l,N 
RII  ,  Jl  *C.O 
CCNTTMJE 
CONTI  ,‘!U  : 

00  3  1*1,7 
00  9  J*  1..N 
8 PS  (  *  ,  J)=»0.0 
CONTINUE 
CONTINUE 

DEFINE  THS  CHARACTEP  ARRAY  OF  PROFESSORS  LI3I 
LIST!  D.'WASHHLFN,  ALAN  R  .• 


L  1ST)  6  )*  1  ESARY  ,  JAMES  0.* 

LIST!  7  )  *  *FI  SCHESCK  ,  PAUL  S.’ 
LIS7<8)**  FCIRR  r  ST ,  R.  N EAGLE  • 
LIST(9)**GAFFCP0,  JACK* 

LIST)  10)  **GA  VS  F  ,  DC.NALD  P.* 

LIST!  11  )  *  *HARTMAN  »  JAMES  K.  ' 
LIST!  12)  **HOIVIK  ,  THOMAS  H.' 
LI5T( 13) ••HOWAPD  ,  GILBERT  T.  • 
LIST) 14 ) * ■ HUGHES ,  WAYNE  P.' 

LIST!  1 5  I  * *HUTC HI  NS ,  CHARLES  W.* 
LIST)  lb )*  "JACOBS »  PATRICIA  A.' 
LIST!  17)  *'L»R5CN,  HAROLD  J.' 
LIST!  18)  *'L3WI  S,  FE^R  A.  W.  • 
LIST)  19  ) » 'Ll  NOSA  Y ,  GLENN  F,  ' 

L 1ST!  20 ) *  *MCM AST  EPS  ,  ALAN  W.* 
LIST(21)**MILCH,  PAUL  R  •  ' 

LIST( 22)*'NEIL  ,  DOUGLAS  E.' 
LIST(23)*, PARRY,  SAWUHL  H.' 
LIST) 2  4 )  *  *  P  SR  R  Y ,  F.  MARCH MAN  * 
LISTI25I »'o0nCK,  GARY  K . 1 
LIST  26 )  * ' PQ.T TER  ,  GARY  R.  ' 

L  I  ST  (  27  )  *  'R  £A  0  ,  ROBERT  R.' 

LISTI  28)  *'RICH4RCS,  F.  RUSSELL* 
l  1ST!  29 )  *  •  SA.NDCZ  .  PATRICK  A.' 

L  IST(  30  )*  *SH'J8 ERT  ,  (WUNO  0.' 
LIST!  31) * 'SHUOCE ,  PFX  H.* 

LIST)  32)  *  'SOL  I  S,  APMfNOO  »  .  • 

L  I3T(  33  )  *'SOVc?E!  ON ,  MICHAEL  G' 
LIST!  34)  *'ST:i*ART,  JOSEPH  B.  ' 
LIST!  35  )  » *S  UL  L  IV  AN ,  ”T  MOTHY  J.' 
LIST!  36)  *'TAYLCR  ,  JA*FS  G.' 


/ 


LIST!  37)  » 1 WOCO  ,  R.  K.  VJN* 

U3T(  ja)«'Y==,  JAWS  R.  • 

LIST (39)  *  'ZEHN A  *  PETER  w.* 

03TAIN  7H=  SUBSET  OP  PROFESSORS  SHEW  3Y  EACH  RESPONDENT 

C*1 

DC  1.0 

RE  AO  (1,2014(11 
FORMAT) 2*. A U 
I F  ( A  (  I)  .£Q.'  I*  )  THEN 
3 (C  )  »( 

C<+I 


EXPLAIN  THE  REQUEST  POP  SI  MIL AR IT Y/D I SS IMILAR  I T  Y  DATA 
PR  CM  THE  RESFCNOENT 

CALL  FRrCNS  ( 'CLRSCRN  ') 

WRfF  (6.U) 

FORMAT  IUI 
MR  ITT  (6,121 
FCRMATUXJ 
WRITE (6,  12) 

FCRmaTIUI 

WfiIT=  (  6.2C1 

*OISS'MlLAi'T?UFCRMNf;W  ,eA0V  TC  S',TER  TN  VALUES  F0R  SIMILARITY  OR 

•|Ss^)AL^isfV,:PCNCP'MINATI0N  CF  TW°  FRC’F£S~I3RS  ACCORDING  -0  YCUR  ? 
WRIT- (6, 321 

•ON  OPTsi*!LARMY’cFMASS  SNTfi'°'  I?l  4  VALU?  R£FL-CTrf!C  r-,JR  PERCEPT! 
WRITE<6, S’T 

„F9^A.U2i‘IsAJ*!I,Jf  tPPSCTIVEMSSS  USING  THE  SCALE  PROVIDED.  LIMIT 
*  « w  jr  ‘J rj i r 5  Tu  AT  *  J 
WRITF(6,34I 

DSC1  MAC  PCINT.  FOR  YOUR  CONVENIENCE  THE  SIMIL 

•ARITY  /O  I  S SIMILAR-  •  1 

WRITC=(6,35) 

FORMaTUX.'ITY  SCALE  APPEARS  2ELQW.  FOR  EXAMPLE,  IF  YOU  PSRCSIVL- 
•PPCFT  .  3PC*N  ANO'l 
WRITE  (6,361 

05i  W2PY  SIMILAR  IN  teaching  EFFECTIVENESS,  YCU 

♦HIGH*  C iUCwC  '£*) 

WRIT"  (6,37) 

PAIR  A  SCORE  0F  2.2  TO  REFLECT  A  HIGH  C5GF3 

•E  OF  jI*ILARITY.') 

WRITE  (  6  ,  All 
FORMA T< 1X1 
WRITE(6,42) 

FORMA  T  I  T20  ,  •  I - -I - - - - - ► - +- - -I - - - |  •) 

WR  ITS  (6,43) 

F CPMAT ( 120 , ' l '  ,T2S,'2',T30,'3',T35,'4',T40,'5',T45,,6',T50,'7',T55 
• , ' 8*  , i 63, '9* 1 
WRIT* (6, 44) 

FORMAT!  T2Q, '  :  •  , T60 , '  :•  I 
WRITE  l  6,45) 

FORMAT!  T 18,  •  VERY  •  ,T 59 , 1  VERY •  ) 

WPIT^  (6,46) 

FORMA  7( 117, 'SIMILA0' ,T56,  'DISSIMILAR' ) 

WRITE  (6,47) 

FORMAT) IX) 

WRITE  (6,49) 

FORMAT (IX) 

WRITT(6,53) 

FORMAT  (  no, 'TO  CONTINUE,  DEPRESS  THE  LETTER  R.M 


c 

c 

c 

c 

52 


C 

C 

c 

c 

6  C 

50 

65 

66 
67 


C 

C 

C 

70 

ac 

90 
100 
110 
120 
130 
140 
1  50 
160 
170 


171 

172 

173 

l  74 


THE  RFA0E.S  VOVA.NCES  TO  CCING  PAIRWISE  COMPARISONS  OF 
P7CFESSCRS 


READ!  5,521  Y 
FORMAT?  VI) 


00? ING  ANO  ALL  PAIRWISE 
A 


IF(Y.Nt.'R')  Tl-E’! 

GG  7  >3  4  g 
ELSE 
eNO  IF 

oo  so  i*i,ci 

Z»I+L 

OC  60  w.c 

CALL  SAI,.  <!,J,F,C,e,LIST) 

THE  SUBROUTINE  PAIR  CONTROLS  t 
C3MPAR I SCNS  FOR  SIMILARITY  DAI 

CCNTf  NUj 
CCNTINU: 

C  ALL  FR  f  CMS  (  ’  CLR.SCRM  •) 

WRI  »  I  6  *66) 

FORMATUX) 

WRITE (6,67) 

FORMA  TUX) 

WRIYT  (6, 70) 

WRITE  (6,80) 

WRIT®  (6,50 
WRJTTU, ICO) 

WRIT?  (6,110) 

WR!T?(6,120) 

WR ITT  (6,  130  ) 

THE  PESPCN’O^mt;  ARE  BRIEFED  ON  THE  BIPOLAR  SCALES 
WRIT:  (6, 1401 

.?9c*ir<i5zi8v,*Rr:  N0W  interested  in  collecting  additional  infcrmat 

•  ICN  ON  CTHcR* ) 

,FOTiJpTBIPaA5*!C~3  C?  YriUR  PRCFE330SS  ‘,N0  TH£  COURSES  'HEY  TAUGH". 

pC^ATIU.;5CALE3  US  20  FOR  THIS  PART  OF  THE  SURVEY  AR^  ST  MI L*  R  IN 
•Oc5Ii,'l  rc  THE  *  ) 

^FORMAT  1 1  X^SIMJLARITY/DI  SSI  HILARITY  SCALE 

FORMA  Y  j  IXi' 0?cf  FA'  “  -  -  -  - 

•RECUIRS3  CNF  CFU 
FORM,,  T(IX,'FQLR  INTEGER  RESPONSES 
•MORE  'HAN'I 

FgflMATdx.'ONCE,  YCU?  RESPONSE  SHOULD  REFLECT 
*HT  TO  yTL  8Y' ) 

FPRMATl IX.'THAT  PROFESSOR. 

WRI TH  ( 6 , 150 ) 

FORMA+UX) 

WRITE (6, 160) 

FORMA T( l X , ' PLEASE  TYPfc  THE  LETTER  R  TO  CONTINUE.') 

READ!  5,  170  Y 
FCRMAT(U) 

IF(Y.Ne.'R')  TH=n 
GO  yq  a; 

ELSE 
tND  IF 

CALL  FB  T CMS  I'CLFSCRN  •) 

OC  L80  (»1, C 
WRITE  (6,1711 

FORMAT)  IX) 

WRIT=  (6,172) 

FORMA  TUX) 

WRITE (6, 173) 

FORMA  TUX, 'THIS  SCALE  OPALS  WITH  THE  TIMEFRAME  IN  THE  CURRICULUM  W 
•HFH  YOU  TOOK  THE') 


PLEASE  LIMIT  RESPONSES 
L  PLACE  EXCEPT  FOR  The  FIRST  BIPOLAR  SCALE  WHICH 
IF  YOU  HAVE  HAD  ONE  PP.OFFSSOP 
THc  LAST  COURSE  TAL'G 

LET  US  BEGIN. ' ) 


WR  tTe  (6,174) 

FORMA  T(1X, 'COUPS' 


TAUGHT  BY  THE  PROFESSOR  LISTED.  FOR  “HIS  SCALE 
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175 

176 

177 

178 

179 
161 
182 
163 

m 

185 

186 
187 

1  90 
161 

2  CO 


162 

16* 


163 

180 

151 

152 

153 
15* 
195 
156 
197 

158 

159 

2a 

2  C2 
2C3 


•ENT"  OS  LY  a  Mr  ■ ) 

WRI7e(6,l7il 

FOUR  V  'LUFS  (1,2,3  OR  4).') 

FORMA  T  (  l  X  ) 

WRIT*;-  (6,177) 

FORMA  T (  ix  | 

W  R  ITr  (6,173) 

FGRMA7{  1X1 
WRITF ( 6, 179) 

FORMAT  ( IX, T20,*  I - - - 

MR!  TP  (6,  181) 

cnnoA  r  i  i  v  -*  i 


•I  •) 


FORMA  7 (IX, "20  ,  *1*  ,7 35, ‘2*, 7  50  ,' 3', 765,* A*) 
WRI  T7  (6  * 182  ) 

"  '  *  ,T35,*s*  ,T50,*S*,765,*  :*) 


FORMAT!  (X, *20 , 

WRITS  (6,1831 
FORMA T  (  T  12  ,4  (  2X  , 1  CCUnSc  T AUGHT’ )  I 
WRITC(6,  18') 

*7TH**0R  '  JTH*  )°  *  2ND  *  ,T  3 1 ,  '3RD  OR  4TH • , T46, ' 5TH  OR  6TH',T61,' 

WR ITF (6,185) 

FORM*  T  (  T9  ,4  ( 3X  » '  CLARTES  •  )  ) 

WRITS  (6,  186) 

FORMAT  (  IX  ) 

WRITS  (6,187) 

FORMa f ( IX) 

MR  ITT  (6,l90ILrST(e(I)) 

FOR-n  ntX.’SMTER  YOUR  RESPONSE  FOR.  * ,  IX ,  A20 ,2X  , '  < - •) 

I  M)«0 

RFAO(  5,2CC,e!(C*l62>  EPS  (1  ,B(7  )  ) 

FORMAT (F 2. 2) 

IF(3PS(l,e(  I)  l.CT.4.0)  THFM 
GC  T0  162 
6t.Se 
SRC  I® 

TFIIVO.  IC.GI  OC  TC  163 
WRIT  "  (  5 , 164 1 

FCRMATUX,1  INCCFRrCT  INPUT,  PLEAS2  ENTER.  AN  INTCGeF.  VALUE*  , 

*  „  .FRjW  l  TU  4.') 

REV) 1 .10  5 
REWfNO  5 
GO  TO  i6l 
CCNTI  NUi 

CALL  FRrCMS  (  'CLRSCRf,  «) 

CCNTTflU* 

OC  210  t«l,C 
WRIT  (6,  151) 

FORMAT ( IX  I 
WRITE  ift,  1921 
FORMA  *  (  l  X  I 
WR  ITE  {  6 «  193  I 

£2?S£7il P0UO.irj6  SCALF  CSALS  WITH  CLASS  SIZE.*) 
wh I  (  6  •  154 1 
FORMAT  (IX) 

WR  I  Tc  (6,  1551 
FORMA  T (  IX) 

WHITER,  1561 
FORMS  T(  T2C,  •  |-- 
WRITF (6, 197) 


•|*> 


»t25.*2*  ,T30,'3',T35f*4*,T40,'5,,T45,'6,,T50,*7,,T55 

m  ,  '  a  *,  r  ovi ,  9*1 
WRITE  (6,1531 
FORMAT!  f  25,' ;  •  ,T55  ,  *  s  * ) 

WRITE  (6,  1991 

WRITE { 6  rfc*  J VEPY  SMALL* ,T5 1, 'VERY  LARGS') 

£CR^T?f|lJ  class  SIZE*, 751, 'CLASS  SIZE') 

WRITE  («,,202) 

FORMAT! IX) 

W RI TE  (6 , 203  1 
FORMAT!  IX) 
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220 

231 


230 


232 

234 


238 

233 

210 

211 

212 

213 

214 
213 
216 
217 
213 

219 

221 

222 

223 

224 

250 

251 

2  60 


WRITT  (6,220)  LISTtem  1 

£22-jr..U*,,5‘iTER  7Cur  RESPONSE  FCS',IX,A20,2X,'<- 

Uun  i  i  '  HJ  ; 

f  N0»0 

i^2i?5^iC;fNn*232,  3PS(2, 8(111 
r  Cr“A T i c  3  m2 ) 

iPtBPS(2,e(in.G'.9.c>  *xen 

epS(2,j(in-8P£(2,8in  i/io.a 


2ND  I  F 

IP(J DM0. "Q.0  I  GC  TC  233 
WRIT  (  i  ,234  ) 

.  W?*WSYWC2*KeT  I25HTt  PL£AZ=  SNTSR  AM  KP-SCSR  VALUE' , 
4«i-:ii,23S)' 

FORM  AT  (|x, 'VALUE  FRCM  1.  a  TO  S.O.'l 

rewind  ; 

GO  *0  221 
CCNTT  NUj 

CALL  FR  f CMS  ('CLTSCRN  •) 

CONTINUE 
DO  24  0  l»l,C 
MBIT';  (6, 211) 

FCPMT(IX) 

WRIT- (6.212) 

FORMA  T( ix) 

WHIT- (6,213) 

l5^^;i^LF-RV9WI',G  SCALC  OSA«-S  WITH  WHETHER  THE  C  OU  P  $  c  WAS  '1 
WRIT?  (6  .214) 

formatux, 'applied  xr:  nature,  n 

WRIT"  (6,215) 

FOPHAT(lX) 

WRITr:  (6,216) 

FORMAT (IX) 

FORMaI  (  f  IgT*  (■ 

WR1T2  (6.2181 


-!■) 


lOtCXOI 

F0RMaTlT2C,;i'  ,T25,'2' , T3C,'3 •  ,T35 ,'4' ,T40,' 5 ' ,745 , '6* ,T50» '7 ' ,T55 

WRITE  (  6,2191 
FORM«T( r2«,':< ,TS«,':') 

WRITE (6, 221) 

£3"2£TLrl^!.,.tPp,-i€!3'  »T50, 'THE  earn  CALM 
WRI : - ( 6,222) 

FCRMAt  (  f  22,  •  CCURS2  •  ,T53,  •  COUP.Sc*  ) 

WRIT-  (6,223) 

FGRMAf(lX) 

WRITE  (6,224) 

FORMAT! IX) 

WRITE (b, 250 )L  I  JT( e< I ) 1 

PORMJTI  Lx,,5Nf.1a  YCUR  RESPONSE  fcr»,  1X,A20,2X,'< - •) 

CuNTi  i U  2 

INO-O 

REAg(5,26C,SN0*252)  9PS(3,8(I)1 
FORMA : (-3.2) 

IF(RPS(1,E(II).GT.9.0)  them 
aP5(3,)(I)  )  -8  FS  (3,8(1)  )/  10.0 

fko  ;  F 

IF(  I’JO.  =C.O)  GC  TC  253 


2  5L 

254 
2  58 

253 


WRIT  ; ( 6 , 254 ) 

0  FORMAJtiX,'  jSCORRgCT  I^P'JTj  PLEAS"  ENTER  AN  INTEGER  VALUE', 

FORM  ATI  IX,  '  VALU':  FROM  1.0  TO  9.0.') 

RcwI'IO  5 
R?W!NO  5 
GO  TO  251 
CONTI NU: 

CALL  FR  r CMS  ( *  CLR3CRN  •) 


88 


240 

261 

262 

263 

26* 

263 

266 

267 

263 

269 

271 

272 

273 

274 

280 

234 

230 


295 

257 


258 

296 

270 

281 

262 

251 

2  52 
233 

283 

284 

285 


CC NTT  MU: 

00  270  t*l,C 
WRITF(6,261 ) 

FORMS  m*) 

WRIT?  (6,262) 

FORMS  7  (IX) 

WRtr  (6,263) 

forms  rax, 'This  scale  reflect;  how  difficult  a  grader  you  perceive 

*0  THF  PICFOSSaF  TC* l 
WRIT-  (6,264) 

FQRMsTt  1 X, ’BE  .  ' ) 

WRIT-  (6,265) 

FORM* T( IX) 

WRITE (6,266) 

FORMS T( IX) 

HR  ITT  (6,267) 

FCRms  T  ( r  20 ,  •  I - * - w - -I - 1 - m - - - - - 1  ■) 

I  TP  (6  2fcd } 

FORMS’!  Flo. * l*  »T2S  ,  *2' , T30.  '  3  *  ,735 ,  *  4*  .740  ,  •  5  •  ,T45  ,  *  6*  ,750 ,  •  7  <  ,755 

WRIT'1  16,269) 

FORMuT  I  T25,  • :  •  ,755,  *  •"  1 
WRITE  (6,271) 

format  up. 'easy*. 754, »  haro*) 

FORMA  T<  lljUGRAOCR*  ,753. '  GRAO  -R*  ) 

WfiTTT  (6,273) 

FORMAT  (IX  ) 

WRIT-  (6.274) 

FORMS  T I  IX) 

MR!  TP  (6.2  SO  )L  1ST  ( E  ( I ) ) 

FORM*  T  I  1  X  ,  'SNT*2  YCL-'  RESPONSE  FOR  '  ,  IX  ,  A20  ,2X  , '  < - *) 

CONTI  DUE 
IAC*0 

READ! 5.25C.SNQ-2SS)  aPS(4,8C)) 

FORMA  "  i  F  3  «2  ) 

IF(3PS{  »  ,8<  I  >  )  .G  .9.0)  THEM 
8PS(4,mn»«»s(4,eu)  )/io.a 
ELSE 
ENO  IF 

IF(IMO.IC.O)  GO  TC  296 

Ufl  FTf*  (  4  idT  j 

FORMAT?  Ix.  ■  IJ1C0PRFCT  INPUT,  SCSAS2  ENTER  AN  INTEGER  V‘.LUC', 

*  »  FRI.M  1  TQ  5  OR  A  REAL*) 

WRITE  (5, 298) 

FORMAT!  IX,  •  WALUC  FROM  1.0  TO  S.O.’l 
REWIND  5 
REWIND  5 
GC  XQ  254 
CQNTINU- 

CALL  FRrCMS  CCLPSCRN  •) 

continue 
oc  30 a  i*i, c 

WRIT?  (6,281) 

FORMS,*  ( IX  > 

WRITE  16.282) 

FORMS : I  IX) 

WRITS ( 6, 251 ) 

F  CRMA  T  ( l  X ,  *  TH I  S  SCALE  .MEASURES  THE  PACT  AT  WHICH  THE  CLASS  WAS  T*U 
*GHT.'  ) 

WRITS  (6.292) 

FORMA  T  (  1X1 


WRIT (6,293) 

FORMAT (IX) 

WRIT  (6.263)  , 

FORMA  T  (T2C,  •  I - 1 - * - * - - 

FORMS  ft  ulV'  1*  »T25  ,  *2*  ,730,  *3  *  ,735  ,'4*  ,T40,*5*  ,T45,'b*  ,7  30  , «  7»  ,7  55 
*,•81,76), *9* ) 


, ,  9 I  9J,  ,  j 

WRITS  (6,285) 

FORMAT!  125,  •  :  '  ,755,*  s  *  I 
WRITE (6.286) 
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FCRMAT! 123.' SLCW* ,T54,'  FAST*  I 
WOI1"-- (  6,287) 

FORMAT ( r 23, *P ACE *,T54,'P ACS') 

WRITS  16,238) 

FORMA  Til  A) 

WRIT*  (  6, 289) 

FORMA  T(  1  X  I 

WRIT:  (fc,310)LI<T(6im 

FORMAT  a  X,'ENT?R  YOUR.  RESPONSE  FOR  * ,  IX, A20.2X ,  '  < - •) 

conhno: 

1N0*0 

READ!  5 ,  3  20,c.N0*322J  9PS  (5 ,3(  I  ) ) 

F QRMA  T  ( 1 3,2) 

IF(BPS(3»8(  l)  1  .GT  .9  .3  1  tHFM 
8PS  ( 5 ,2 i I ) ) *  e  FS ( 5 , B ( I )  1/10,0 
EL  S3 
ENC  IF 

IFIINO.EC.O)  GC  TC  323 
WR ITS  (  6 , 324 ) 

FORMATUX,  * ’MCCRRECT  INPUT,  PLEASE  ENTER  AN  INTEGER  VALUE*, 

*  '  FR1P  I  TO  9  OR  A  RFAL’ ) 

WRITE  (6 ,328) 

FORMAT  (  IX,  '  VALUE  FROM  1.0  TO  9.0.*) 

REWI.10  5 
R|WIND  5 
g£  TO  121 
CONTT  NU  2 

CALL  FRTCMS  t  *  CLRSCRN  •) 

CCNTT  NU: 

DC  330  1*1, C 
WR ITS (6,301) 

FORMAT!  IX) 

WRITE  (6,  202) 

FGSMATIIX) 

WRIT1 (6,203) 

FORMAT  (IX, 'THE  FALLOWING  SCALE  MEASURES  THE  AMOUNT  OF  EFFORT  REQUI 
•RED  6Y  rhF  STUC2NT* ) 

WRIT  (6,315) 

FORMoTUX, "OUTSIDE  OF  CLASS.') 

WRITE  (6,304) 

FORMAT! IX) 

WRITS  (6,  305  I 
FORMAT!  IX) 

WRITcI  6.3C7)  *“  ’  * 

FORMAT!  r  20,'  1*  ,T2! .  •  2*  ,  T30,  *  3  •  ,T35  ,  •  4*  ,T40,*5*  ,T45  ,  '  6'  ,T50  ,  *7*  ,T55 
»,*8*,T63,*9*  I 
WRIT-  (6,308) 

FORMAT (r2s,'S',T55,'S*) 

FORM^-ftf  19?1  LITTLE  85  PORT  •  ,T50 ,  'MUCH  EFFORT*) 

FORMAT-t  ri7i*  RcCUIRED  OUTSIDE*  »T48»  ' REOU IP.cO  OUTSIDE*) 

WRIT1  (6,  3161 

F0RMAT(T|3, J CLASS *,T53, 'CLASS'  ) 

FORMA  T  ( I X  ) 

WRITS  16,313) 

FORMAT! IX) 

WRITE  (6,3  40  ILT  ST  (Eli)  ) 

FORMAT! IX, 'ENTER  YCUR  RESPONSE  FOR’ , IX, A20 ,2X, * < - *) 

CONTI NUi 

rtFAO?  5,  )5Q,cNO*352)  3PS(6,B(I)) 

FORMAT  IF  2.2) 

I  F(  3P  S  (  6  ,ei  I )  )  .GT  .9.0)  then 
BPS!  6, 3  ill  )*8PS  (6,8(1) 1/10.0 
ELSE 
ENO  t  F 

IF!  INO.-C.O)  GC  TC  353 
WRI”: (  6 , 254) 


354 


3  58 

353 

330 

331 
3  32 
333 

3  35 
3  36 
337 
3  33 

3  39 

341 

342 

343 

344 

345 

3  70 
38L 

380 


382 

384 


3  88 

3  83 
380 

386 


410 


F0R‘4aT(  IX,  •  INCC'SEC1'  INPUT  ,  nEATE  ENTER  AN  1,'ITSGSR  VALUE*  , 

•  '  ea  JN  1  rc  ■)  CS  i  RrAL'  1 
WSITcts.353) 

FORMAT! IX, 'VALU*  FRC  4  1.0  TO  5.0.*! 

Rgwr'IO  5 
RCWVIO  5 
GC  TO  1J i 
CONTI  MU  * 

CALL  ERTCVS  {  *  CLASCRN  •) 

CONTT  MU: 

00  360  l*l,C 
WRITE ( 6 1  331 1 
FORMA’1! IX) 

WSITT  (6,  322) 

FCRMA  T (IX  I 
writ:  {  6,3331 

FORMA T(IX»‘THE  LAST  SCALE  ASKS  TO  WHAT  EXTENT  THE  COURSE  HSU  SO  UP 

•  ON  THE  3R£REQU  IS  ITES  .  * ) 

WRIT: (6,335) 

FORMAT< IX) 

WPITF  (6,336) 

FORMA T( IX) 

WRITS (6, 337) 

FORMAT)  r 20,  *  I - * - - - ’ - - - - - - - - - 1  •) 

WRITE  (6,  2281 

FORM*  T(  T20,  *  1  *  ,T2  5,  *2*,T30,  *  3  *  ,T35  , »  4*  ,  T40  ,  *  5  *  ,T45 ,  •  6*  ,T50  ,  •  7*  ,T  55 
•,  *a*,T63  ,*9» ) 

WRIT"-  (6,2391 

FORMAT (.25,* : • ,T55, • :  *  I 

WRIT?  (6,341) 

FORMAT|  r^.'CCUSTE  RELIED ',T49,  'COURSE  RELIED') 

WRITE  16,342) 

F0RMftT(T2l, 'LITTLE  G.M'  ,T5  1,  'HEAVILY  ON') 

WRITS  1 6, 343  > 

FORMAT !  1 19, '  PP.EPsCLI  SITES  '  ,T49 ,  •  PPSRECU  IS  ITCS  *  I 
WRITE  (6,  344) 

FORMAT  iu  ) 

WRI1T  (6,245) 

FORMAT! IX) 

WRITE  16.370  )L  1ST- 1  etD), 

FORMA  TUX.'-NTSp.  VGLR  RESPONSE  C0P  '  ,  IX,  A20 ,2X ,  '< - •) 

CONTI MUs 
I  N0»0 

PFAO(  5,180, £'10*3321  BPS(7,81I)) 

FORMAT  1*3,2) 

IPOPSlf  ,8!  I)  >  .GT.9.0)  THEM 
BPS!  7, a U) l»aPS (7,8(1) 1/10-Q 

f  NoS  F 

IF(IMO.-C.O)  GC  TC  383 
V*R  lT  E  1 4  ,384 ) 

FORMAT( iX, < 'NCORRCCT  INPUT,  FLEAS*  ENT  SR  AN  INTEGER  VALUC', 

•  '  FS1H  1  TO  9  OR  A  REAL ' ) 

WRIT  2(4  ,388 ) 

FORM AT ( IX , 'VALUE  FRCM  1,0  TO  9,0. •> 

REWINO  5 
REWI‘10  5 
GO  TO  381 
CONTI  tIUS 

CALL  FRfCMS  ( 'CLPSCRN  '! 

continue 

CALL  FRrCMC  CCLPSCRN  •) 

WRITE  (6,386) 

FORMAT!  IX) 

WRITS (6,387) 

FORMAT!  IX) 

THE  OAT  A  IS  COLLECTED  ANC  STORED  II)  EACH  ARRAY 
00  400  t*l,M 

WRIT* (2 ,410) (P(I ,J) ,J-1,NI 
FORMAT! 20F4.1) 
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4cu 

4  15 
417 


4  30 
420 


LC 

li 

25 


35 
3  C 


40 

42 

43 

44 

55 

46 

50 

45 


61 

62 


63 


70 


C0NT-'  JU: 
warn  (2,415) 

F  QRMA  T  {  l  X  ) 

WRIT:  (2,4171 

forma  T(  IX) 

DC  42  0  (-1,7 

hr  it s ( >  ,430  K  epr. (i  ,ji  ,j  =  i,;)) 

FORm.\’(  2CF4.  l  ) 

CCNTI  ;iU3 
STCP 

fno 

REQUESTS  FES  STMt LA RITY/CISSI HILARITY  SCORE  S  ARF  MADE  HERE 

S  LAYOUT  INF  PAIRC  ,J,P,C,8,LIS7) 

CCHHlM/iRRCOH/INC 
Integer  i,c,at29)  ,j,k,l 
REAL  »( 39,39 1 
O-ARaCTER»20  LI ST  (39 ) 

CALL  PR  7 CMS  ( ' CLRSCRN  • ) 

HR  ITT  (6,  1C) 

WRITM6.U)  L!ST(E(I)) 

f  ORW.  T  (  f  2  ,*  °Ly  ASr-  FNTTR  IM  YOUP  VALUE  FCR  SIMILARITY  D  T  ."ACHING  = 

•  FFECT IVEN6S3  CCMPAKIEG • ) 

FCRMAT! IX, A20.1 X,* AGAINST  THE  FPOFESSOF  INDICATED  bY  THE  ARROW. ' ) 
IF(J.LT.C)  THEN 
K-J-H 

WRf'F  (6  ,25  )  L  T  ST  (  8  ( J  ) ) 

FORMAT! 2CX, 12C.2X, •< - 1 ) 

CC  30  L-K,C 
WRI  TCI  6,35)  L IS  T (6  t  L)  ) 

FOP  MAT (20X.A20I 
CCNTIfJJg 
FLSE 
K*J 

HR  IT  F I  i  ,40  ILlSTIBtK)) 

FORM  ATI  2CX  ,A2C  ,2X  ,  ‘  < - • ) 

f-NC  IF 
WRIT=(6.42) 

FORMAT l  r2Q,  <  | - - - ► - - - - - - - - - - - |  •) 

WRIT"  (6.42) 

FCSMAT  l  720.' I  ■  ,T25,.'2',T30,  •  3  •  ,T35,*  4'  ,740 ,  •  5  ' ,  T45  ,  *  61  ,T50  ,  *7’  ,T  55 
*,  '3*,  T6J ,'9'  ) 

WRITE  (6.44) 

FGRMaT  (  720. '  S  '  ,T 60,  •  s  •  1 
WRI~  (  c,  55) 

FORMAT!  r  18,  'VERY*  ,T5'J, '  VERY' ) 

iFORmS  ■ftriT’i'SIFILAF'  ,T56,  ‘DISSIMILAR*  ) 

CONTINUE 
I  NO-0 

RcAOl 5,  iS.FND-61)  P(8(I),8(J)) 

FCPM.%t(S3.2) 

IF(R(S(I),3(J)).GT.9.3)  TH*jN 
R(8(  I),81J))»P(3(I)  ,  3(  J  I  )  / 10  .0 

Ifio'l  f 

IF(TND.EC.O)  GC  TC  70 

FCPMATt  lx?  •  INCORFFCT  INPUT,  FlEASF  ENTER  AN  INTEGER  ViLU'S 

*  •  FR  IN  1  TC  9  CR  A  REAL’ ) 

HR  IT  E  (  6  ,63) 

FCRMATI  IX,  •  VALUE  FF.LH  1.0  TO  9.0.') 

RFHINO  5 
PEHI  10  5 
GO  T0  5C 
CONTINUE 
PFTIF  'I 
6NC 

F1RCR  HANCLING  SUSR  CUT  I NF  USFO  TO  HELP  WF.HO  OUT  INCORRECT 
RESPONSES 
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